TI811X PSP PCI Express Endpoint Driver User Guide

TI811X PCI Express Endpoint Driver User Guide

Linux PSP

Introduction

TI811X device has PCI Express hardware module which can either be configured to act as a Root Complex or a PCIe Endpoint. This document caters to the Endpoint mode of operation and describes the Driver needed to configure and operate on TI811X PCI Express device as Endpoint.

This driver is referred as EP Driver and has to be loaded on TI811X EP running Linux kernel from PSP package. It is expected that a userspace application would use this driver to configure the PCIe module and do transfers with peers.

Refer Package Contents and Download section for details about getting all associated components.

Scope of the Document

At present, this document covers following areas:

• Details about the EP Driver
• Summary of applications and kernel modules provided as sample
• Build configuration and instructions
• Execution

Assumptions and Conventions

Following points/prerequisites are assumed throughout this document:

1. For using PSP release 04.07.00.01 (v2.6.37_TI811XPSP_ENG5_REL), the kernel tarball from release package is extracted and build environment is set up as described in User Guides packaged with respective releases.
2. All kernel modules/driver source and header file paths are relative to the kernel source directory where the package is extracted, except otherwise stated.
3. References to kernel modules provided as sample assume that the patches for the same are applied to kernel extracted.
4. References to sample applications assume the application patch is applied in an empty directory created by the user on the build host.
5. All application files are referred with path relative to the above directory.

Terminology

• PCIe - PCI Express
• EP - PCIe Endpoint
• RC - PCIe Root Complex

Package Contents and Download

The EP Driver package consists of main driver along with sample test applications and supplementary drivers (custom samples).

• The EP driver is available in kernel source for TI811X release 04.04.00.02 (int_ti81xxpsp_04.04.00.02).
• The sample applications for RC side as well as EP side are in attached tarball. Click here to download. Further discussions assume that this file is downloaded on build host as epdrv_sample_apps.tar.gz.
• EDMA sample driver for TI81XX EP side and Sample driver for TI81XX RC side is provided as patch file here to be applied to kernel snapshot from int_ti81xxpsp_04.04.00.02

Supported Setups

Currently following setups are tested:

• TI811X RC <-> TI814X EP
• TI814X RC <-> TI811X EP
• TI811X RC <-> TI811X EP

For details about doing these setups, please refer following documents:
TI811X PCIe RC Driver User Guide
TI811X PCIe Boot Driver User Guide

Not Supported / Known Issues

• Must follow the sequence of loading various components as described in steps in Execution section below, other sequences are not yet validated and may not work
• Interrupt reception and handling on EP side from RC is not supported (this any way is non-standard PCIe feature).
• MSI generation at EP side and handling at RC side is supported for throughput (THPT) mode tests.
• Running RC application on EP and vice verse may lead to issues and should be avoided.
• ioctl values for various drivers currently overlap with existing drivers in kernel. Particularly, debugging tools such as strace will treat the ioctls for the drivers as one of the existing ioctl already supported in kernel: eg., TI81XX_EDMA_SET_BUF_INFO is reported as SNDCTL_DSP_CHANNELS

EP Driver Summary

The EP driver controls the TI811X Endpoint device it is running on.

It creates a device node named "ti81xx_pcie_ep" under "/dev" to enable application to open and configure the device through character device interface.

Driver Calls

open() Application have to first open the device to get the file descriptor for calling subsequent functions.

ioctl() Various IOCTLs can be used by the application to access the PCIe hardware on local TI811X EP.

mmap() MMAP interface can be used by the application using EP driver to map the reserved memory for PCIe as well as the h/w window of 256MB starting from 0x20000000.

close() Close the driver and cleanup outbound mappings.

IOCTLs Supported

TI81XX_SET_INBOUND Set inbound translation for a given BAR on TI811X EP.

TI81XX_SET_OUTBOUND_SIZE Set the size for outbound window. The same size is applicable to all windows configured.

TI81XX_SET_OUTBOUND Create outbound window for specified address. This address has to be specified by the application such that a valid PCIe address is generated over the PCIe link.

TI81XX_ACCESS_REGS Read of write the specified PCIe module register - this could be from PCIe module's application space or local configuration space.

TI81XX_SEND_MSI Send MSI targeted to specified address.

TI81XX_GET_PCIE_MEM_INFO Application can use this to query the size and physical address of the reserved memory on TI811X EP for PCIe transfers. This memory space can then be divided by the application across different BARs (for inbound mapping) and to perform CPU/EDMA read/write transfers from EP. To access this memory in userspace, the application can use mmap facility provided by the EP driver (for CPU transfers) or by EDMA sample driver for EDMA trnasfers.

TI81XX_GET_OUTBOUND_STATUS Return a 32-bit mask inidicating available outbound windows. A value '1' for specific bit position indicates corresponding window in free and not enabled for outbound.

TI81XX_CLR_OUTBOUND_MAP Clear outbound mapping for specified address. Corresponding window will be marked as free.

TI81XX_GET_INTR_CNTR Get the current count of received MSI interrupts. This count is cumulative from the initialization of opening of driver. Presently not tested.

TI81XX_GET_BAR_INFO Returns the information associated with particular BAR such as address assigned to it (by RC) and size of the BAR.

EP Driver Build Configuration

The default TI81XX kernel build doesn't enable EP driver. Follow steps below to enable and build EP driver:

NOTE: Currently, EP driver can be enabled only if PCI Bus Support is disabled in kernel configuration. This is done to avoid any conflict when mistakenly running EP driver on a Root Complex.

Using the correct kernel

• Ensure that the kernel from downloaded from ti81xx-master branch is used and extracted on build host.

Ensuring default kernel configuration

• Configure the kernel with default options. E.g., for TI811X EVM,

make ARCH=arm ti811x_evm_defconfig

Enter kernel configuration menu

make ARCH=arm menuconfig

You will be presented with menu such as below:

   General setup  --->
   [*] Enable loadable module support  --->
   -*- Enable the block layer  ---> 
      System Type  --->
       Bus support  --->
   ...
   ...
   Device Drivers  --->

Disable PCI Bus Support

• Remove PCI bus support, which will in turn remove PCIe Root Complex Driver.
• Scroll till "Bus Support" listed above using DOWN arrow and press SPACE to enter the option menu:

   [*] PCI support
   [*] Message Signaled Interrupts (MSI and MSI-X)
   [ ] PCI Debugging
   < > PCI Stub driver
   [ ] PCI IOV support
   < > PCCard (PCMCIA/CardBus) support  --->

Disable PCI support by pressing 'n' key on "PCI support" option. This will automatically disable other dependent configurations.

   [ ] PCI support
   < > PCCard (PCMCIA/CardBus) support  --->

Now press RIGH arrow to highlight "Exit" and press ENTER to return to initial menu.

Enable EP Driver
Scroll with DOWN arrow key till "Device Drivers" option and press SPACE. This will present following menu:

   Generic Driver Options  --->
   < > Connector - unified userspace <-> kernelspace linker  --->
   ...
   ...
       Character devices  --->

Scroll till "Character devices" and press SPACE, which will show a menu similar to following:

   -*- Virtual terminal
   ...
   ...
   < > TI81XX PCIe Endpoint Driver (NEW)

Using DOWN arrow key, go till PCIe Endpoint Driver option as shown above.

Press 'y' to build the driver into kernel

   ...
   <*> TI81XX PCIe Endpoint Driver

or 'm' to build as loadable module.

   ...
   <M> TI81XX PCIe Endpoint Driver

NOTE: At present, only modular build is supported/validated so it is recommended to build the driver as module.

Use RIGHT arrow key to highlight "Exit" and press ENTER successively till you are prompted to save the kernel configuration. Select "Yes" and press enter to exit the configuration menu.

Building EP Driver

Once the EP driver is enabled in kernel build configuration as described in earlier section, you can build it as follows:

Building statically with kernel
In this case, rebuild of kernel is required and driver will be built into kernel

make ARCH=arm uImage

Building as loadable module

NOTE: Even when loading the EP driver as module, you need to build the kernel at least once and make sure that it is used for EP before loading the module otherwise the EP driver may not get loaded successfully. In addition, to enable reserving memory at boot time and for successful EP driver build, PCIe EP driver must be selected in build configuration before building the kernel.

Build the kernel wil PCI support disabled and EP driver selected to be built as module (refer earlier section for configuration):

make ARCH=arm uImage

Build kernel modules

make ARCH=arm modules

This will build EP driver module located inside drivers/char as:

drivers/char/ti81xx_pcie_epdrv.ko

Resource Reservation

The EP drive expects a contiguous block of memory reserved at boot time to facilitate applications to do mmap when doing data transfer using PCIe. The desired size should vary depending upon the application requirement.

By default there is no memory reserved for EP driver unless a kernel commandline argument "pcie_mem" with non-zero value specifying size of the memory to be reserved is passed.

NOTE1: PCIe EP driver must be included in kernel configuration to be able to reserve memory for PCIe.

NOTE2: The EP driver doesn't require any memory to be reserved for its own functionality, but this memory is required to provide mmap regions mapping to PCIe BARs (inbound transfers) and/or EDMA regions (outbound transfers) to allow application to perform PCIe communication and transfers.

An example for reserving 16 MB of RAM for PCIe EP, append following to kernel bootargs:

pcie_mem=16M

Note that, though the user can specify the size of memory to be reserved for PCIe, the start address of this chunk is not guaranteed to be same across different booting of the kernel and may vary due to parameters such as different boot arguments, kernel configuration and changes in kernel. The EP drivers provides an IOCTL to enable application (using EP driver) to query the physical base address and size of the reserved PCIe memory block.

WARNING: If PCIe memory is not reserved during boot time, the EP driver would still load successfully but the application on EP might fail if it expects reserved RAM available to transfer data using mmap or DMA. To ensure that the driver had reserved memory available, the application must use TI81XX_GET_PCIE_MEM_INFO ioctl to check if reserved memory size fits application's requirement.

Prerequisites on initialization

EP driver expects that the PCIe h/w on (local) EP is already initialized before the driver is initialized. This includes:

• PCIe subsystem is set as PCIe Endpoint (h/w default)
• PCIe link is up. This means SerDes, PLL is setup to provide 2.5 GHz clock from 100MHz reference and PRCM is setup to enable PCIe module and LTSSM on.
• Root Complex has detected this EP and allocated resources such as memory BARs, interrupt required for its operation.

Following the "Execution" steps described in this document automatically ensures all of the above since PCIe boot operation of EP takes care of them.

In case you want to use a non-PCIe boot mode, all of the above needs to be taken care of before starting the EP driver. Refer TRM for PCIe h/w initialization details. Note that to ensure that the RC detected and configures this device, the PCIe initialization should be done at early stages during boot up.

Inbound Mapping

EP driver, on its own, doesn't create inbound mapping for any BAR. It is the application's responsibility to allot a contiguous region from PCIe reserved memory to create inbound mapping using IOCTL TI81XX_SET_INBOUND.

Note: It is recommended but not forced on the application to use the region from PCIe reserved memory to create inbound mapping. It may as well use a different mechanism to reserve memory (outside of EP driver) and pass the physical address etc to EP driver for setting up inbound translation.

Outbound Mapping

Unlike inbound, outbound mapping doesn't directly depend on the PCIe memory reserved, since it is internal masters (Cortex A8 CPU or EDMA) who need to be aware of (or provided with) memory to be used as source/destination for outbound and is independent of the PCIe and EP driver.

In this case also, a contiguous space for transfer is needed and it may be convenient to use a part of memory already reserved for PCIe. For example, in case the memory reserved is 8 MB, designate buffers in the lower 4 MB to carry out EDMA reads/writes to/drom.

NOTE: EP driver, on its own, doesn't provide any EDMA transfer functionality and it is the application's responsibility to set up outbound transfer parameters such as outbound mapping (using IOCTL TI81XX_SET_OUTBOUND), determining source/destination address for transfer, etc. before proceeding to use EDMA driver.

EP Driver Files

drivers/char/ti81xx_pcie_epdrv.h
drivers/char/ti81xx_pcie_epdrv.c

Kernel Files/Functions Added/Updated

• arch/arm/plat-omap/devices.c
  • Adds mechanism to reserve RAM for PCIe EP operations at boot time with size

specified by "pcie_mem" boot argument.

• arch/arm/plat-omap/include/plat/common.h
• arch/arm/plat-omap/common.c
  • Reserve memory during boot

Sample Components

The various sample components to utilize the EP driver on TI81XX EP are attached to this page.

These include:

• EP side: Sample kernel module for EDMA
• RC side: Sample kernel module for TI81XX RC which provides RC side application with access over PCI
• RC & EP side: Sample Applications

The details are covered in individual sections below.

EDMA Kernel Module

This module is provided as a sample driver to access EDMA functionality from user space. The main purpose of this driver is to enable the EP side application to initiate EDMA transfers to the peer over PCIe. Note that since this is a sample driver, it is not integrated into kernel and needs to be applied as patch to the kernel source from release.

Driver Files

drivers/char/ti81xx_edma.h drivers/char/ti81xx_edma.c

Driver Calls

Please refer driver files and EP application code for reference.

IOCTLs supported

Please refer driver files and EP application code for reference.

Building The Module

Apply the attached patch 0001-TI811x-PCIE-Added-EDMA-and-RC-sample-drivers-for-J5E.patch by copying it to kernel source directory. Ensure that the kernel is already configured for TI81XX EP as mentioned earlier.

The download link mentioned earlier has the patch in gzipped format so it needs to be extracted to get the patch file. You can then use 'patch' utility to apply the patch as:

# gunzip 0001-TI811x-PCIE-Added-EDMA-and-RC-sample-drivers-for-J5E.patch.gz
# patch -p1 0001-TI811x-PCIE-Added-EDMA-and-RC-sample-drivers-for-J5E.patch

NOTE: The above step directly modifies the drivers/char/Makefile to build this sample driver always as loadable kernel module.

Build module with following command

make ARCH=arm modules

The module will be built as

drivers/char/ti81xx_edma.ko

RC Kernel Module

This sample driver needs to be applied to the RC kernel. This means it must have PCI bus support with MSI enabled.

Note: At present, only TI814X/TI811X RC is supported. Also, to apply this patch cleanly, the kernel extracted earlier (for building EP driver) needs to be used.

Driver Files

drivers/char/ti81xx_pcie_rcdrv.h drivers/char/ti81xx_pcie_rcdrv.c

Driver Calls

Please refer driver files and RC application code for reference.

IOCTLs supported

Please refer driver files and RC application code for reference.

Building The Module

NOTE: The above patch directly modifies the drivers/char/Makefile to build this sample driver always as loadable kernel module when PCI MSI (CONFIG_PCI_MSI) support is enabled.

Ensuring default kernel configuration

• Configure the kernel with default options. E.g., for TI811X EVM,

make ARCH=arm ti811x_evm_defconfig

Build the kernel for RC

make ARCH=arm uImage

Build RC module

make ARCH=arm modules

The module will be built as

drivers/char/ti81xx_pcie_rcdrv.ko

Sample Applications

The attached tarball epdrv_sample_apps.tar.gz contains RC and EP side sample applications which use various kernel modules described above to communicate and perform data trnasfers between them across PCIe.

Currently the EP application requires minimum 16MB of memory reserved for PCIe transfers, thus, the kernel command line must be passed pcie_mem=16M at least. If the memory reserved is less, then the application will exit.

Files

./Makefile
./lib
./lib/Makefile
./lib/ti81xx_ep_lib.c
./lib/ti81xx_mgmt_lib.c
./lib/ti81xx_pci_info.c
./lib/ti81xx_trans.c
./include
./include/debug_msg.h
./include/ti81xx_ep_lib.h
./include/ti81xx_mgmt_lib.h
./include/ti81xx_pci_info.h
./include/ti81xx_trans.h
./application
./application/makefile
./application/ti81xx_ep_app.c
./application/ti81xx_rc_app.c

Building Applications

Create an empty directory and extract the attached tarball epdrv_sample_apps.tar.gz into that directory.

E.g., assuming the directory created is named 'apps' and eprv_sample_apps.tar.gz is copied to it,

# cd apps
# tar -xzvf eprv_sample_apps.tar.gz

Build the applications for TI811X RC and TI811X EP using following command

# make KERNSRC=<absolute-path-to-TI811X-kernel-source>

Remember to pass correct path to kernel source where the kernel tarball mentioned earlier is extracted.

The above command will build EP and RC applications at following paths respectively:

application/ep_app
application/rc_app

In case the RC kernel source is different than EP kernel (e.g., when using PC -> TI811X setup), the RC and EP side application components need to be built separately in two steps as below example:

# make KERNSRC=<absolute-path-to-RC-kernel-source> rc
# make KERNSRC=<absolute-path-to-TI811X-EP-kernel-source> ep

You may use make clean in between to force rebuild of the library files which are common for RC & EP. Note that this will also delete the application built in earlier step, so ensure it is copied elsewhere.

To clean the built files, you can use

# make clean

Note: Currently the DISPLAY and THPT mode of applications are supported. The other mode - INTEGRITY mode is not supported.

Execution

Following sequence needs to be executed to run the demo with above components (commands with 'EP#' to be executed on EP side while those with 'RC#' to be executed on RC side):

• Ensure the desired PCIe setup is done - in this case, we assume TI811X <-> TI811X setup.
• Power on TI811X EP
• Power on TI811X RC
• Boot EP from RC (refer Boot Driver User Guide)

Note: It should be possible to skip this step and instead have EP booted with following procedure.

On TI811X EP side, build the U-boot.

• Apply the patch by copying it to U-Boot source directory.

The download link has the patch in gzipped format so it needs to be extracted to get the patch file. You can then use 'patch' utility to apply the patch as:

# gunzip 0001-TI811x-PCIE-Added-support-for-PCIe-EP-configuration.patch.gz
# patch -p1 < 0001-TI811x-PCIE-Added-support-for-PCIe-EP-configuration.patch

• Build MLO (U-Boot-MIN for SD boot) and u-boot (U-Boot 2nd stage with ENV on NAND)(Refer U-boot user guide).
• Copy the built-images(MLO, u-boot and uImage) in the SD Card and Boot EP.

On TI811X RC side.

1. Navigate to that directory and load the boot driver

RC# insmod ti81xx_pcie_bootdrv.ko  

Continue with the steps to load the EP side drivers mentioned below.

• Make sure that at least 16 MB of PCIe memory is reserved by appending pcie_mem=16M (or more) to the 'bootargs' to the EP kernel. You may also need to ensure that the 'mem' argument passed to kernel ensures sufficient memory is available.
• Ensure that EP Driver, EDMA driver and EP sample application are present in a directory on EP filesystem.
• On TI811X RC side, make sure that the RC driver and application are present in working directory.
• Navigate to that directory and load the boot driver

RC# insmod ti81xx_pcie_bootdrv.ko

• Once shell is available on EP side, navigate to the directory containing the above files. Load EP Driver and EDMA sample driver:

EP# insmod ti81xx_pcie_epdrv.ko
EP# insmod ti81xx_edma.ko

• After successful load of these modules, run EP application:

EP# ./ep_app

• You may need to add executable permission to the application before above step:

EP# chmod a+x ep_app

• The EP application will not wait for communication with peer.
• On TI811X RC side, make sure that the RC driver and application are present in working directory.
• Navigate to that directory and load the RC driver

RC# insmod ti81xx_pcie_rcdrv.ko

• Run RC application (again, you may need to run chmod a+x rc_app before this step)

RC# ./rc_app

Since these applications are built with DEMO mode, they will start communicating each other and do data transfer in either directions using CPU (read/write from RC) and EDMA (read/write from EP).

During execution of the applications, on RC and EP consoles, you would see chunks with 'C', 'D', 'L' characters. The meaning is explained below:

on RC, data represented by 'C' is sent by memcpy from EP
on RC, data represented by 'D' is sent by EDMA from EP
on EP, data represented by 'C' is sent by memcpy from RC
on EP, data represented by 'L' is data that has been read by EDMA from a buffer on RC populated by application on RC.

• To exit the DEMO, terminate RC and EP apps - this can be done by pressing Ctrl + C combination at shell on both RC and EP. Note that there are no log files generated for DEMO mode.

Getting Throughput from EDMA transfers

For throughput measurements, MSI interrupt from EP to RC are used for control communication.

• You can also build the applications in throughput mode to get the throughput data for EDMA transfers from EP (read/write).
• To set this mode, build the applications by passing MODE=THPT as argument to 'make' command.
• Assuming we are building the applications for TI811X (RC) <-> TI811X (EP), using following commands on build host in EP sample applications directory

# make clean
# make KERNSRC=<path-to-TI811X-kernel-source>  MODE=THPT

• This will build the applications ep_app and rc_app inside application directory. Transfer them to RC and EP as mentioned earlier and set as executable.

• It is assumed that the required modules are loaded on RC and EP after fresh boot as described earlier to proceed application execution
• On TI811X EP, start ep_app followed by starting rc_app on RC.
• In the example setup, following messages were seen

• On TI811X EP

THPT calculated in TX direction is : 323.028381 MBPS
Executing test case THPT RX (RC read from EP)
THPT calculated in RX direction is : 339.973450 MBPS
Executing test case THPT TX/RX (simultaneous read & write from EP)
THPT calculated in RX direction is : 139.243942 MBPS
THPT calculated in TX direction is : 138.903961 MBPS
Total thpt calculated tx+rx is:  278.147888 MBPS

• On TI811X RC

root@ti811x-evm:/home/user/ti811x_pcie# ./rc_app
***Forcing polling (interrupt communication disabled, RC will not generate interrupt)