Compiler Validation

Compiler Testing Challenges

There are unique challenges in testing compilers. This is due to several key issues:

1. Testing is difficult due to the complexity of the compiler since the source includes many algorithms involving instruction scheduling, register allocation, software pipelining, vector optimization, and so on.
2. The test input domain is virtually infinite. Even if some inputs are erroneous or meaningless, the compiler must correctly handle those inputs without fatal internal errors, and must report reasonable messages back to the user.

In addition to these general compiler challenges, there are additional challenges in testing the TI compiler.

1. The TI compiler must be tested across multiple targets and multiple option combinations.
2. The TI compiler supports multiple execution platforms.
3. The TI compiler supports a long-lived code base, in some cases extending to 25 or more years.

To overcome these challenges, the compiler command-line shell tool, the C I/O run-time libraries, and TI command-line simulators allows for batch processing of compiler testing. TI has developed extensive automation to handle the batch testing of the compiler.

Robustness Mechanisms

There are several mechanisms in place to address compiler robustness at TI. Some key points:

1. Re-use: The compiler supports a minimum of 6 different target architectures. There is extensive re-use of compiler technology across these targets in each compiler release. Some details:
   • About 60% of the code base is target-independent.
   • For any given compiler about 90% is target-independent.
   • On average each line of source code is re-used 6 times.
   • Test code is also re-used. Most tests developed for one compiler are leveraged for use with all targets.
2. Peer Reviews: Development of compiler technology is handled through several review cycles before being added to the source base.
   • Each new feature requires a functional review.
   • Each new feature requires a separate design review.
   • Prior to a code review, new features require a code overview where implementation details are analyzed and decided.
   • Each change to the code base, even minor, is review by at least one other developer. Code reviews are performed on-line using [Code Collaborator].
   • Check-ins to the code base are prevented until the reviews are completed.
3. Analysis Tools: Several analysis tools are used both during development and during testing.
   • [Purify]: Each release is checked for memory leaks, buffer-overruns, heap corruption, dangling pointers, etc.
   • [PureCoverage], [Gcov]: Coverage data is collected on each release and correlated by module, tool, and target.
   • Coverage data is stored in a database for tracking and history.
4. Automation: The compiler build and validation process is entirely automated. The input to a compiler release validation includes a test plan and the final output of the validation is a tar file of all validation results as well as a summary report. Some automation details:
   • The build and validation automation controls nightly builds and validations, as well as all release validations.
   • [LSF] is used to leverage thousands of hosts to run the validations in parallel.
   • Nightly builds/validations are run on thousands of hosts with a 12-16 hour running time.
   • Release builds/validations are run on thousands of hosts with a 48-72 hour running time.

Compiler Validations

TI compiler validation is performed at 3 different stages.

1. Prior to check-in: A "pre-commit validation" must successfully pass in order for a change to be allowed into the code base. This pre-commit validation contains a sampling of test suites designed to provide almost full coverage.
2. Nightly validation: This tests a nightly build which incorporates all committed and approved changes from the previous day. The nightly validation must successfully pass in order for any new changes are allowed into the code base. This validation includes the complete test suites, expanded option combination testing, and random option combinations. In addition, the nightly validation will test interactions between all source updates from the previous day.
3. Release validation: A release validation includes all complete test suites and each suite is tested with extensive option sets. These option sets will include hundreds of combinations. A release validation will compile over 200,000 individual executable, self-checking test cases.

Compiler Test Suites

Commercial Test Suites

The following commercial test suites are run during the compiler validations.

1. Plum Hall C and C++
   • The Plum Hall Validation Suite for C
   • Suite++: The Plum Hall Validation Suite for C++
2. Perennial C and C++
   • CVSA Latest ISO C Validation Suite
   • C++VS ISA C++ Validation Suite
3. ACE C and C++
   • SuperTest C/C++ Compiler Test and Validation Suite
4. CodeSourcery C++
   • C++ ARM ABI Test Suite
5. Dinkumware C++/STL Validation
   • Proofer for C++/STL Validation
6. Nullstone C Validation
   • Nullstone for C
   • Optimization tests
7. GNU C and C++ Testing
   • Torture Tests

In-House Validation Suites

The following in-house developed test suites are run during the compiler validations.

1. Kernels
   • These are key loops that are checked for performance and correctness.
2. Applications
   • These are common benchmark applications such as gsmefr, jpeg, mpeg, etc. Included in this group are industry performance benchmarks. Also included are customers applications which are run for performance, code-size, and/or correctness.
3. Regression Tests
   • The code for each bug report is added to the regression suite.
4. Unit Tests
   • These are black and white box tests of specific functionality.