StarterWare Test Framework

Document License

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 United States License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/us/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.

StarterWare Automation Test Framework Overview

StarterWare Automation Test Framework is a test environment used to test the StarterWare example applications. The purpose of this document is to provide an overview of using the StarterWare Automation Test Framework for validating StarterWare platforms.
Testing using the framework will be carried out at standalone example applications level. The configuration ability of the example application defines the number of peripheral configurations that may be exercised.
The test framework is not SoC OR platform specific but is developed and tested for EVM/reference h/w only. However, test environment needs to be created for every new platform.
The tests shall have the following pass/fail criteria:
i. Execution of examples without Error messages.
ii. Non-occurrence of a system hang or a crash.
iii. All selected configurations work properly (where applicable).

This Framework includes the following components:

NOTE
1> Though the frame work is generic to support multiple processors of the AM/OMAP family, support may be in a phased manner. The test frame work support for the processors is mentioned in the corresponding processors release notes. The frame work is currently tested for only one instance of the test setup and evaluated for AM335x EVM.
2> Testing certain applications through StarterWare Automation Test Framework is not a substitute for human testing. This is with reference to applications like McASP or Raster where human presence is a mandatory requirement to perceive the audio or the image being displayed.

• TestLink. Test Link is an open source test management tool.
• Test Execution Engine (TEE). This is the software running on the Test-PC that is responsible for executing and validating the test on DUT and interfacing TestLink for receiving test requests and reporting test results. Current version of TEE is supported to function on Windows PCs only.
• Device Under Test (DUT). This is the target hardware put to test.

The tests can be triggered from TestLink by configuring the TEE aptly (Scenario-1 of Figure: StarterWare Test SetUp; detailed in later sections) OR can be triggered directly from TEE, independent of Testlink. The Tests can be triggered by BuildForge ( as depicted in Scenario-2 of Figure: StarterWare Test SetUp) also. However, this feature is currently not available.

Hardware Requirements

• DUT and its power supply
• USB Remote power switch
• Serial Cable to connect DUT to Test PC
• Test PC with Windows XP OS
• TestLink Server

Software Requirement

• Perl
• Java Runtime Environment
• STAF
• TEE
• TeraTerm
• TestLink (Not needed when triggering tests from PC without TestLink)

Supported OS

• Test PC environment

• Windows XP Service Pack 3

• TestLink

• Windows and Linux

SetUp

SetUp Hardware

Setup the hardware test environment as in below figure. Make sure the power switch of the DUT is ON.
Test Scenario-1
i. Testcases/test suites are created on Testlink.
ii. Register TEE to Testlink.
iii. Tester can log into TestLink and initiate execution of either each test case or in a bunch test cases.
iv. TEE parses, executes, analyzes (automated mode)/accepts (semi-automated mode) test results and results are updated to TestLink.
v. The results of the tests can be viewed in TestLink.

Test Scenario-2
i. Testcases/test suites are created on Testlink.
ii. Register TEE to Testlink.
iii. Tester can log into BuildForge and initiate build of the release.
iv. BuildForge builds the package.
v. The binaries are copied to the ‘Test PC’.
vi. BuildForge logs into TestLink and initiate execution of either each test case or in a bunch test cases.
vii. TEE parses, executes, analyzes (automated mode)/accepts (semi-automated mode) test results and results are updated to TestLink.
viii. The results of the tests can be viewed in TestLink.
This scenario is not supported in current version of the test framework.

Test Scenario-3
i. Testcases/test suites are created in the ‘Test PC’.
ii. Initiate test from ‘Test PC’.
iii. The results of the tests can be viewed in results file locally created.

SetUp Software

Install - Third party software

• Install Active Perl

 http://www.perl.org/get.html

• Install Win32/SerialPort.pm library from below link. Copy the 'lib' contents of SerialPort to 'lib' folder of Perl.

 http://search.cpan.org/~bbirth/Win32-SerialPort-0.22/lib/Win32/SerialPort.pm

• Install XML parser: XML.pm library from below link. Copy the 'lib' contents of XML parser to 'lib' folder of Perl.

 http://search.cpan.org/~rjray/RPC-XML-0.76/lib/RPC/XML.pm

• Install Java (java_ee_sdk-5_01-windows.exe) in default path.

 http://www.oracle.com/technetwork/java/index.html

• Install STAF (STAF348-setup-win32.exe) in default path. Indicate the right version of Perl being used with STAF.

 http://staf.sourceforge.net/getcurrent.php

• Install TestLink Automation Framework

 http://sourceforge.net/projects/testlink/

Install/unpack - StarterWare_TestFramework

NOTE
Make sure the install/un-packed path is free of spaces as in 'Documents and Settings'. The absolute path leading to the TEE base folder will be referred to as "<TEE_INSTALL_DIRECTORY>" hence forth.

Update the necessary environmental variables and paths in the source and configuration files.

• Modify ...\STAF\bin\STAF.cfg to reflect *your* settings; Refer to 'STAF.cfg' field details under 'Directory Structure and files' section.
• Modify below files to your test environment. Refer to corresponding file details under 'Directory Structure and files' section.

• 'StarterWareTB.pl'-> Present in "<TEE_INSTALL_DIRECTORY>"
• 'config.yml'-> Present in "<TEE_INSTALL_DIRECTORY>\framework\testlink_interface\parser"
• 'StarterWare_TL_TEE_Link.pl'-> Present in "<TEE_INSTALL_DIRECTORY>\framework\testlink_interface\parser"
• 'downwardTranslatorStarterWare@1.xsl'-> Present in "<TEE_INSTALL_DIRECTORY>\framework\testlink_interface\translator"
• 'StarterWareTB.pl' and '<TEST_INPUT_FILE_NAME>.txt' files with environment setting for TestLink based Test execution.

Please set the environment setting for TestLink based Test execution for the above files.

• Set Environment Variables for all above Installed Packages if not set.

• Right click on MyComputer -> Properties -> Advanced -> Environment Variables -> System Variables -> Path
• If the tools are installed under'C:' drive. The path mey be updated by adding: C:\STAF\bin;C:\Perl\bin\;C:\Perl\site\bin;C:\Sun\SDK\bin;C:\Sun\SDK\lib;C:\Sun\SDK\jdk\bin;

NOTE
The PATH settings should be set as installed in the local machine of the user.

Test Execution and Results

TestLink based:

• Open TestLink page using below provided link

http://testlinksvr01.india.ti.com/testlink/index.php

• User need to create account in Testlink. Request for ‘TI-Lead’ privileges for ‘StarterWare’ test project for your user id.

• Select Test Project 'StarterWare' and Test Plan 'am335x_evm'.
• To know your 'devkey' or 'Personal API access key' (mentioned in config.yml file), to change your password, E-mail id change etc select the 'Personal' link.
• To create 'Builds / Releases', select 'Builds / Releases'

• Select 'Create button'
• Add a title to the Build/Release
• Add below two lines under 'Description'

• platform =<PLATFORM_USED>;
• assign_to_tees = <TEE_NAME> on <TEST_MACHINE_IP>;

• For E.g. platform: am335x_evm
• TEE named: 'StarterWare@x0093626'
• Test PC IP: 192.168.247.107

• Format is given in the below Image
• Save the Changes made.

• Creating/Adding New TestCase: Select 'Specification' from TestLink Home-Page.

• Navigate and select Test Project: StarterWare. Here new test suites may be created.
• Navigate and select Test Suite: am335x.
• Existing test cases can be seen under the test suite.
• Select 'Create Test Cases' button to view the 'Create Test Case' page.
• Add 'Test Case Title'.
• Select "Execution type : Manual"
• Select "Test importance : Medium"
• Add "Test execution engine: <YOUR_TEE_NAME>". Note, this name should match with the one in STAF.cfg file under ...\STAF\bin and that declared in 'Builds / Releases'.
• Add "Test script or logic: NA"
• Add "Required hardware assets: dut1 = ["<platform>"]". User should not update <platform> in this field.
• Add "Application parameters: <testCaseId> <binaryName> <timeOut> <scriptName>". User should provide the test specific details in between <>. When no custom script is needed, '0' should be used.
• Select 'Create' button to create a new test case.
• Select the test case, select the "Add to Test Plans" button and select the test plan.

• Select 'Execute' from TestLink Home-Page.
  
• Select the "TestSuite" OR "TestCase" and select "Execute and Save Results" option.
  
• For results, select 'Results' and then 'Test result matrix' from reports and metrics.
  

TEE based:

• Start STAF: open command prompt and execute “stafproc”.
  
• Navigate and open <TEE_INSTALL_DIRECTORY>\framework\test\platform\<PLATFORM APPLICABLE>\<TEST_INPUT_FILE_NAME>.txt file.
  
• Add test case/line in the file in below specified format for <TEST_INPUT_FILE_NAME>.txt file.
  
• Open command prompt to <TEE_INSTALL_DIRECTORY>.
  
• Execute "StarterWareTB.pl" present in <TEE_INSTALL_DIRECTORY>.
  
• Wait for the tests to complete.
  
• "TCResults.txt" file present in the <TEE_INSTALL_DIRECTORY> folder contain the test results.
  

Troubleshooting

• Testcase triggered from TestLink internally triggers "StarterWareTB.pl" file. Incase of any errors, test may be triggered from TEE using "StarterWareTB.pl" file, to narrow down configuration issue.
• In certain cases, ttermpro.exe was not being triggered OR not identified as valid executable on the test PC. In such cases, copy the ttermpro.exe and ttpmacro.exe files in to ...\STAF\bin.
• In case Test triggered from TestLink exits prepaturely, cause of the issue may be identified by adding some probe points in 'StarterWare_TL_TEE_Link.pl' by those familier with perl scripting:

• Since the scripts terminates soon after the error, add sleep after "&ExecuteTestCase;" call. Check the cause for error.
• TestLink exports the testcases in XML format to the test PC, which may be seen under ...\STAF\data\STAF\tmp. this folder typically has files named

• xxx_downwardTranslatorStarterWare@1.xsl
• xxx_upwardTranslatorStarterWare.xsl
• <TEE_NAME>_txed_request_***.xml : This file constitutes of the exported test cases from TestLink.

• Some STAF commands:

• To Start STAF, open command prompt execute “stafproc” .
• To end STAF, open command prompt execute “staf local shutdown shutdown”.
• To see the current tee, open command prompt execute “staf 172.24.128.156 resmgr list”.
• To turn ON the power switch 1, execute “staf 192.168.247.107 usbremotepower on 1”
• To turn OFF the power switch 3, execute “staf 192.168.247.107 usbremotepower on 3”

• The DUT need to be power cycled after every test for UART BOOT. If the boot is not going through, check if the DUT is power cycled OR not.

Directory Structure and files

Under the <TEE_INSTALL_DIRECTORY>, files 'StarterWareTB.pl' and 'STAF.cfg' and a folder 'framework' can be seen.

• StarterWareTB.pl - This is the test logic responsible for getting the to be executed test list, execute and analyze the tests and store the results.

- This is the file that need to be executed when triggering tests from test PC. - Update the existing IP address with your test PC IP address. Search for the line that goes as: '"staf 192.168.247.107 usbremotepower ' - Update the 'binaryFilePath', 'configFilePath', 'logFilePath' and 'scriptsFilePath' details with correct details. E.g: Replace "$binaryFilePath = "framework/test/platform/evmAM335x/binaries/";" with "$binaryFilePath = "framework/test/platform/evmAM1808/binaries/";" if the plaform is changed from am335x EVM to am1808 EVM. - Update the existing Power Switch port number in this file, with the Power Switch port number to which your DUT is connected.

NOTE
This file must be updated to user test environment.

• STAF.cfg - This is the configuration file for STAF execution engine. Access permissions to TestLink/dispatcher, TEE registration details and RPS service control details are updated with local test frame work settings. User may update the:
  

- Absolute path leading to the test TEE folder. E.g.: If the TEE is installed in "c:/StarterWare_TestFramework", replace all the instances of "D:/perlTools/automation/v5" with "c:/StarterWare_TestFramework".
- Update the test PC IP. E.g. for test PC IP 172.24.128.101, replace "Staf/Config/Machine=192.168.247.107" with "Staf/Config/Machine=172.24.128.101"
- Update the “trust machine” details with IPs from where STAF may be invoked
- Update the “#Power Controller Service” details with “port number” controlling the RPS
- Update the USELIB path where ‘powerControl.pm’ perl library is available (available with TEE at "<TEE_INSTALL_DIRECTORY>\framework\staf_services\powercontrol\”)
- Update TEE name: starterware@x0093626 to what ever needed
- Update “D:/perlTools/automation/v5” in the reference .cfg file need to be updated to where the ‘framework’ folder and ‘StarterWareTB.pl’ files are present
- Update “hwassets” shall be updated with the platform being supported.

NOTE
This file must be updated to user test environment.
This file is provided for reference and shall be copied into ...\STAF\bin folder.

• framework - This folder contains three folders: staf_services, testlink_interface and test. These sub folder constitute of various configuration files, test execution and analysis script files, test binaries files, etc, all needed for automating the test execution.
  

• staf_services - This folder contains files providing STAF services.
  

• teeadapter.jar - Java run time libraries needed by STAF. Doesn't need any changes for varying test setups.
  
• powerControl/powerControl.pm - Perl library file for controlling the RPS. Doesn't need any changes for varying test setups.
  

NOTE
This file must be changed for STAF to use the power control library.

• testlink_interface - This folder will contain all the scripts and files needed to interface with TestLink.
  

• parser - This folder will contain all the interface scripts and files needed for test trigger from TestLink. 'listTestCases.txt' is an exception and need to be updated for TEE based test trigger.
  

• StarterWare_TL_TEE_Link.pl
  

- Update the absolute path leading to 'config.yml' in this file. Search for the line that goes as: my $config = do{local(@ARGV,$/)=" - Update the TEE name (starterware\@x0093626) as set for your environment in this file. Search for the line that goes as: my @teeInFile = `staf local - This perl scripts is responsible for below activities

• Parse the test request received from TestLink into 'listTestCases.txt' file
• Trigger the 'StarterWareTB.pl' logic.

NOTE
This file must be updated to user test environment.

• listTestCases.txt - This file has the test case data in the format: "testCaseId" "binaryName" "timeOut" "scriptName".

NOTE
This file is updated by 'StarterWare_TL_TEE_Link.pl' based on the test cases triggered from TestLink. User need not make any changes in this file.

• config.yml - This file has the environment data needed in the format: pathFrameworkBase, projectId, testLinkServerIp, devkey, testLinkServerUrl, upLoadTranslatorName, downLoadTranslatorName, resultsFileName, tl_login, tl_password.
  

- Configure '<TEE_INSTALL_DIRECTORY>' to your environment
- Configure devkey if login is different. The 'devkey' details may be obtained from TestLink Personal details, under 'Personal API access key' field.

NOTE
This file must be updated to user test environment.

• translator - This folder will contain three scripts used by the framework.
  

• automation_results_template.xml - This file is a dummy result file needed by the STAF framework. User need not make any changes to this file.
  

NOTE
This file need not be changed for any test environment.

• upwardTranslatorStarterWare.xsl - This file is a file needed by the STAF framework to update results one by one after test execution. User need not make any changes to this file.
  

NOTE
This file need not be changed for any test environment.

• downwardTranslatorStarterWare@1.xsl - This file is a file needed by the STAF framework to download test requests. This file need not be changed except for the field value of "params".
  

User need to update this file with the absolute path of 'StarterWare_TL_TEE_Link.pl' in the line: "params">START COMMAND perl XXX/StarterWare_TL_TEE_Link.pl

NOTE
This file must not be changed for any test environment.

• test - This folder contains all the files/scripts of the platforms that are being tested. Any new platform needing support should have its own copy of sub folders present under "<TEE_INSTALL_DIRECTORY>\framework\test\platform" folder.
  

• teraterm_config - This folder contains scripts to use TeraTerm for serial communication.
  
• platform

• <PLATFORM APPLICABLE>
  

• <TEST_INPUT_FILE_NAME>.txt - This file has the test case data in the format: "testCaseId" "binaryName" "timeOut" "scriptName".

- This file need to be update manually when tests are triggered from TEE.

- Each field is separated by a space and each line corresponds to details for one test-case.

NOTE
This file must be updated to user test environment for TEE based tests only.

• binaries - This folder shall hold the binaries of the test applications and the UART boot loader. UART boot loader binary 'boot.bin' and the test application should be free of headers.
  
• scripts - This folder contains scripts to execute and analyze the tests.
  

• analyze_default.pm - This script file is used when the test app is 'run to completion' type with no user input needed AND test analysis is only based on logs with NO ERROR keyword.
  
• <APPLICATION NAME>\<APPLICATION_NAME>_TC<TESTCASE_NUMBER>.pm - This script file serves as reference in case where the test app needs custom input and custom analysis. The function name should be named <ScriptName>_TC_<TestCaseId>. The text input expected over console is put in a file <ScriptName>_TC_<TestCaseId>.txt. For E.g. for "1 hdr_uartEcho.bin 30 hdr_uartEcho_TC1.pm", uartEcho_TC1.txt should be the text file with input.
  
• <APPLICATION NAME>\<APPLICATION_NAME>_TC<TESTCASE_NUMBER>.txt - This file contains the data send over UART. Data in this file is organised in the ordered expected by the test application.

NOTE
User may update the script logic to suit their test case. For just inputing the data as in the .txt file, user may take reference of 'uartEcho_TC1.pm' file provided with the package and update the subroutine to suit the testcase.

• logs - This folder contains logs of the tests executed.