TI81XX PSP PCI Express Endpoint Driver User Guide

TI81XX PCI Express Endpoint Driver User Guide

Hemant Pedanekar

Introduction

This document is applicable to DM816x/AM389x, DM814x/AM387x and DM813x family of devices referred hereafter as TI816X, TI814X and TI813X respectively.
Descriptions common across these device families use the term TI81XX/ti81xx.

TI81XX devices have PCI Express hardware module which can either be configured to act as a Root Complex or a PCIe Endpoint. This document caters to the Endpoint mode of operation and describes the Driver needed to configure and operate on TI81XX PCI Express device as Endpoint.

This driver is referred as EP Driver and has to be loaded on TI81XX EP running Linux kernel from PSP package. It is expected that a userspace application would use this driver to configure the PCIe module and do transfers with peers.

NOTE: This document is applicable to PSP GIT kernel available from Arago git post PSP 04.00.01.13 release for TI816X and 04.01.00.06 release for TI814X or kernel from release 04.00.02.14/04.01.00.07/04.04.00.01 or later for respective platforms. Jump to Package Contents and Download section for details about getting all associated components.

Scope of the Document

At present, this document covers following areas:

• Details about the EP Driver
• Summary of applications and kernel modules provided as sample
• Build configuration and instructions
• Execution

Following items will be incorporated in future versions of this document:

• Internals of driver and sample code details - for now, please refer the respective source code.
• More details about setups
• Design and expected/recommended use cases of the EP driver
• EDMA usage details
• Throughput data

Assumptions and Conventions

Following points/prerequisites are assumed throughout this document:

1. If using PSP release 04.00.01.13, the PSP kernel tarball package from Arago git is extracted and build environment is set up as described in PSP User Guide associated with 04.00.01.13 release package.
2. Else, if using PSP release 04.00.02.14 (TI816X) or 04.01.00.07 (TI814X) or 04.04.00.01, the kernel tarball from release package is extracted and build environment is set up as described in User Guides packaged with respective releases.
3. For TI813X support, the latest kernel source from ti81xx-master branch on Arago is used.
4. All kernel modules/driver source and header file paths are relative to the kernel source directory where the package is extracted, except otherwise stated.
5. References to kernel modules provided as sample assume that the patches for the same are applied to kernel extracted in case #3 above. For other kernel versions, some manual merging of patches may be
6. References to sample applications assume the application patch is applied in an empty directory created by the user on the build host.
7. All application files are referred with path relative to the above directory.
8. The term "TI81XX EP" refers to local TI816X/TI814X/TI813X EP where the EP driver instance runs.

Terminology

• PCIe - PCI Express
• EP - PCIe Endpoint
• RC - PCIe Root Complex

Package Contents and Download

The EP Driver package consists of main driver along with sample test applications and supplementary drivers (custom samples).

• Out of these, the EP driver is available in kernel source for TI816X release 04.00.02,14 or TI814X release 04.01.00.07 and onwards. Users of earlier release may download the kernel tarball including first version of EP driver directly from TI81XX Arago tree on ti81xx-master branch from here, though recommended approach is to get kernel source from tip of ti81xx-master branch on Arago to get TI813X boot and EP driver support and also avoid merge conflicts when applying the sample driver patches.
• The sample applications for RC side as well as EP side are in attached tarball. Click here to download. Further discussions assume that this file is downloaded on build host as epdrv_sample_apps.tar.gz.
• EDMA sample driver for TI81XX EP side is provided as patch file here to be applied to kernel snapshot from ti81xx-master branch
• Sample driver for TI81XX RC side is provide as a patch on the same kernel snapshot (after applying EDMA patch).

Supported Setups

Currently following setups are tested:

• TI814X RC <-> TI816X EP
• TI816X RC <-> TI816X EP
• TI816X RC <-> TI814X EP
• TI816X RC <-> TI813X EP

For details about doing these setups, please refer following documents:
TI81XX PCIe RC Driver User Guide
TI81XX PCIe Boot Driver User Guide

Note: There may be some porting effort and code changes required when non TI81XX RC is involved.

Not Supported / Known Issues

• Following warning is observed while building kernel and RC module for TI81XX after applying RC driver patch

warning: 'virt_to_bus' is deprecated (declared at /db/psp_git/users/a0393588/temp/linux-omap-2.6/arch/arm/include/asm/memory.h:222

• This doesn't have any impact on execution and can be ignored

• Must follow the sequence of loading various components as described in steps in Execution section below, other sequences are not yet validated and may not work
• Interrupt reception and handling on EP side from RC is not supported (this any way is non-standard PCIe feature).
• MSI generation at EP side and handling at RC side is validated only for throughput (THPT) mode tests (refer sample application) since 04.04.00.01 release.
• Running RC application on EP and vice verse may lead to issues and should be avoided.
• ioctl values for various drivers currently overlap with existing drivers in kernel. Particularly, debugging tools such as strace will treat the ioctls for the drivers as one of the existing ioctl already supported in kernel: eg., TI81XX_EDMA_SET_BUF_INFO is reported as SNDCTL_DSP_CHANNELS

EP Driver Summary

The EP driver controls the TI816X/TI814X/TI813X Endpoint device it is running on.

It creates a device node named "ti81xx_pcie_ep" under "/dev" to enable application to open and configure the device through character device interface.

Driver Calls

open() Application have to first open the device to get the file descriptor for calling subsequent functions.

ioctl() Various IOCTLs can be used by the application to access the PCIe hardware on local TI81XX EP.

mmap() MMAP interface can be used by the application using EP driver to map the reserved memory for PCIe as well as the h/w window of 256MB starting from 0x20000000.

close() Close the driver and cleanup outbound mappings.

IOCTLs Supported

TI81XX_SET_INBOUND Set inbound translation for a given BAR on TI81XX EP.

TI81XX_SET_OUTBOUND_SIZE Set the size for outbound window. The same size is applicable to all windows configured.

TI81XX_SET_OUTBOUND Create outbound window for specified address. This address has to be specified by the application such that a valid PCIe address is generated over the PCIe link.

TI81XX_ACCESS_REGS Read of write the specified PCIe module register - this could be from PCIe module's application space or local configuration space.

TI81XX_SEND_MSI Send MSI targeted to specified address. Note that the MSI target address is needed only when sending MSI from EP local to remote EP, no target address used when sending MSI to RC from EP since the MSI address is already configured by RC.Presently only validated when sending MSI to RC from EP.  

TI81XX_GET_PCIE_MEM_INFO Application can use this to query the size and physical address of the reserved memory on TI81XX EP for PCIe transfers. This memory space can then be divided by the application across different BARs (for inbound mapping) and to perform CPU/EDMA read/write transfers from EP. To access this memory in userspace, the application can use mmap facility provided by the EP driver (for CPU transfers) or by EDMA sample driver for EDMA trnasfers.

TI81XX_GET_OUTBOUND_STATUS Return a 32-bit mask inidicating available outbound windows. A value '1' for specific bit position indicates corresponding window in free and not enabled for outbound.

TI81XX_CLR_OUTBOUND_MAP Clear outbound mapping for specified address. Corresponding window will be marked as free.

TI81XX_GET_INTR_CNTR Get the current count of received MSI interrupts. This count is cumulative from the initialization of opening of driver.

TI81XX_GET_BAR_INFO Returns the information associated with particular BAR such as address assigned to it (by RC) and size of the BAR.

EP Driver Build Configuration

The default TI81XX kernel build doesn't enable EP driver. Follow steps below to enable and build EP driver:

NOTE: Currently, EP driver can be enabled only if PCI Bus Support is disabled in kernel configuration. This is done to avoid any conflict when mistakenly running EP driver on a Root Complex.

Using the correct kernel

• Ensure that the kernel from downloaded from ti81xx-master branch is used and extracted on build host.

Ensuring default kernel configuration

• Configure the kernel with default options. E.g., for TI816X EVM,

make ARCH=arm ti8168_evm_defconfig

• When building for TI814X EVM as EP,

make ARCH=arm ti8148_evm_defconfig

• For TI813X EP,

make ARCH=arm dm385_evm_defconfig

Enter kernel configuration menu

make ARCH=arm menuconfig

You will be presented with menu such as below:

   General setup  --->
   [*] Enable loadable module support  --->
   -*- Enable the block layer  ---> 
      System Type  --->
       Bus support  --->
   ...
   ...
   Device Drivers  --->

Disable PCI Bus Support

• Remove PCI bus support, which will in turn remove PCIe Root Complex Driver.
• Scroll till "Bus Support" listed above using DOWN arrow and press SPACE to enter the option menu:

   [*] PCI support
   [*] Message Signaled Interrupts (MSI and MSI-X)
   [ ] PCI Debugging
   < > PCI Stub driver
   [ ] PCI IOV support
   < > PCCard (PCMCIA/CardBus) support  --->

Disable PCI support by pressing 'n' key on "PCI support" option. This will automatically disable other dependent configurations.

   [ ] PCI support
   < > PCCard (PCMCIA/CardBus) support  --->

Now press RIGH arrow to highlight "Exit" and press ENTER to return to initial menu.

Enable EP Driver
Scroll with DOWN arrow key till "Device Drivers" option and press SPACE. This will present following menu:

   Generic Driver Options  --->
   < > Connector - unified userspace <-> kernelspace linker  --->
   ...
   ...
       Character devices  --->

Scroll till "Character devices" and press SPACE, which will show a menu similar to following:

   -*- Virtual terminal
   ...
   ...
   < > TI81XX PCIe Endpoint Driver (NEW)

Using DOWN arrow key, go till PCIe Endpoint Driver option as shown above.

Press 'y' to build the driver into kernel

   ...
   <*> TI81XX PCIe Endpoint Driver

or 'm' to build as loadable module.

   ...
   <M> TI81XX PCIe Endpoint Driver

NOTE: At present, only modular build is supported/validated so it is recommended to build the driver as module.

Use RIGHT arrow key to highlight "Exit" and press ENTER successively till you are prompted to save the kernel configuration. Select "Yes" and press enter to exit the configuration menu.

Building EP Driver

Once the EP driver is enabled in kernel build configuration as described in earlier section, you can build it as follows:

Building statically with kernel
In this case, rebuild of kernel is required and driver will be built into kernel

make ARCH=arm uImage

Building as loadable module

NOTE: Even when loading the EP driver as module, you need to build the kernel at least once and make sure that it is used for EP before loading the module otherwise the EP driver may not get loaded successfully. In addition, to enable reserving memory at boot time and for successful EP driver build, PCIe EP driver must be selected in build configuration before building the kernel.

Build the kernel wil PCI support disabled and EP driver selected to be built as module (refer earlier section for configuration):

make ARCH=arm uImage

Build kernel modules

make ARCH=arm modules

This will build EP driver module located inside drivers/char as:

drivers/char/ti81xx_pcie_epdrv.ko

Resource Reservation

The EP drive expects a contiguous block of memory reserved at boot time to facilitate applications to do mmap when doing data transfer using PCIe. The desired size should vary depending upon the application requirement.

By default there is no memory reserved for EP driver unless a kernel commandline argument "pcie_mem" with non-zero value specifying size of the memory to be reserved is passed.

NOTE1: PCIe EP driver must be included in kernel configuration to be able to reserve memory for PCIe.

NOTE2: The EP driver doesn't require any memory to be reserved for its own functionality, but this memory is required to provide mmap regions mapping to PCIe BARs (inbound transfers) and/or EDMA regions (outbound transfers) to allow application to perform PCIe communication and transfers.

An example for reserving 8 MB of RAM for PCIe EP, append following to kernel bootargs:

pcie_mem=8M

Note that, though the user can specify the size of memory to be reserved for PCIe, the start address of this chunk is not guaranteed to be same across different booting of the kernel and may vary due to parameters such as different boot arguments, kernel configuration and changes in kernel. The EP drivers provides an IOCTL to enable application (using EP driver) to query the physical base address and size of the reserved PCIe memory block.

WARNING: If PCIe memory is not reserved during boot time, the EP driver would still load successfully but the application on EP might fail if it expects reserved RAM available to transfer data using mmap or DMA. To ensure that the driver had reserved memory available, the application must use TI81XX_GET_PCIE_MEM_INFO ioctl to check if reserved memory size fits application's requirement.

Prerequisites on initialization

EP driver expects that the PCIe h/w on (local) EP is already initialized before the driver is initialized. This includes:

• PCIe subsystem is set as PCIe Endpoint (h/w default)
• PCIe link is up. This means SerDes, PLL is setup to provide 2.5 GHz clock from 100MHz reference and PRCM is setup to enable PCIe module and LTSSM on.
• Root Complex has detected this EP and allocated resources such as memory BARs, interrupt required for its operation.

Following the "Execution" steps described in this document automatically ensures all of the above since PCIe boot operation of EP takes care of them.

In case you want to use a non-PCIe boot mode, all of the above needs to be taken care of before starting the EP driver. Refer TRM for PCIe h/w initialization details. Note that to ensure that the RC detected and configures this device, the PCIe initialization should be done at early stages during boot up.

Inbound Mapping

EP driver, on its own, doesn't create inbound mapping for any BAR. It is the application's responsibility to allot a contiguous region from PCIe reserved memory to create inbound mapping using IOCTL TI81XX_SET_INBOUND.

Note: It is recommended but not forced on the application to use the region from PCIe reserved memory to create inbound mapping. It may as well use a different mechanism to reserve memory (outside of EP driver) and pass the physical address etc to EP driver for setting up inbound translation.

Outbound Mapping

Unlike inbound, outbound mapping doesn't directly depend on the PCIe memory reserved, since it is internal masters (Cortex A8 CPU or EDMA) who need to be aware of (or provided with) memory to be used as source/destination for outbound and is independent of the PCIe and EP driver.

In this case also, a contiguous space for transfer is needed and it may be convenient to use a part of memory already reserved for PCIe. For example, in case the memory reserved is 8 MB, designate buffers in the lower 4 MB to carry out EDMA reads/writes to/drom.

NOTE: EP driver, on its own, doesn't provide any EDMA transfer functionality and it is the application's responsibility to set up outbound transfer parameters such as outbound mapping (using IOCTL TI81XX_SET_OUTBOUND), determining source/destination address for transfer, etc. before proceeding to use EDMA driver.

EP Driver Files

drivers/char/ti81xx_pcie_epdrv.h
drivers/char/ti81xx_pcie_epdrv.c

Kernel Files/Functions Added/Updated

• arch/arm/plat-omap/devices.c
  • Adds mechanism to reserve RAM for PCIe EP operations at boot time with size

specified by "pcie_mem" boot argument.

• arch/arm/plat-omap/include/plat/common.h
• arch/arm/plat-omap/common.c
  • Reserve memory during boot

Sample Components

The various sample components to utilize the EP driver on TI81XX EP are attached to this page.

These include:

• EP side: Sample kernel module for EDMA
• RC side: Sample kernel module for TI81XX RC which provides RC side application with access over PCI
• RC & EP side: Sample Applications

The details are covered in individual sections below.

EDMA Kernel Module

This module is provided as a sample driver to access EDMA functionality from user space. The main purpose of this driver is to enable the EP side application to initiate EDMA transfers to the peer over PCIe. Note that since this is a sample driver, it is not integrated into kernel and needs to be applied as patch to the kernel source from release.

Driver Files

drivers/char/ti81xx_edma.h drivers/char/ti81xx_edma.c

Driver Calls

Please refer driver files and EP application code for reference.

IOCTLs supported

Please refer driver files and EP application code for reference.

Building The Module

Apply the attached patch 0001-char-ti81xx-Add-EDMA-driver-to-provide-minimal-acces.patch by copying it to kernel source directory. Ensure that the kernel is already configured for TI81XX EP as mentioned earlier.

The download link mentioned earlier has the patch in gzipped format so it needs to be extracted to get the patch file. You can then use 'patch' utility to apply the patch as:

# gunzip 0001-char-ti81xx-Add-EDMA-driver-to-provide-minimal-acces.patch.gz
# patch -p 1 -i 0001-char-ti81xx-Add-EDMA-driver-to-provide-minimal-acces.patch

NOTE: The above step directly modifies the drivers/char/Makefile to build this sample driver always as loadable kernel module.

Build module with following command

make ARCH=arm modules

The module will be built as

drivers/char/ti81xx_edma.ko

RC Kernel Module

This sample driver needs to be applied to the RC kernel. This means it must have PCI bus support with MSI enabled.

Note: At present, only TI816X/TI814X/TI813X RC is supported. Also, to apply this patch cleanly, the kernel extracted earlier (for building EP driver) needs to be used.

Driver Files

drivers/char/ti81xx_pcie_rcdrv.h drivers/char/ti81xx_pcie_rcdrv.c

Driver Calls

Please refer driver files and RC application code for reference.

IOCTLs supported

Please refer driver files and RC application code for reference.

Building The Module

Apply the attached patch 0002-char-ti81xx-Add-driver-for-controlling-TI81XX-EP-fro.patch by copying it to kernel source directory. Ensure that the patch for EDMA sample module is already applied (as mentioned above), this will avoid conflicts.

The download link mentioned earlier has the patch in gzipped format so it needs to be extracted to get the patch file. You can then use 'patch' utility to apply the patch as:

# gunzip 0002-char-ti81xx-Add-driver-for-controlling-TI81XX-EP-fro.patch.gz
# patch -p 1 -i 0002-char-ti81xx-Add-driver-for-controlling-TI81XX-EP-fro.patch

NOTE: The above step directly modifies the drivers/char/Makefile to build this sample driver always as loadable kernel module when PCI MSI (CONFIG_PCI_MSI) support is enabled.

Ensuring default kernel configuration

• Configure the kernel with default options. E.g., for TI816X EVM,

make ARCH=arm ti8168_evm_defconfig

• In case using TI814X EVM as RC, use,

make ARCH=arm ti8148_evm_defconfig

• For TI813X RC,

make ARCH=arm dm385_evm_defconfig

Build the kernel for RC

make ARCH=arm uImage

Build RC module

make ARCH=arm modules

The module will be built as

drivers/char/ti81xx_pcie_rcdrv.ko

Sample Applications

The attached tarball epdrv_sample_apps.tar.gz contains RC and EP side sample applications which use various kernel modules described above to communicate and perform data trnasfers between them across PCIe.

Currently the EP application requires minimum 8MB of memory reserved for PCIe transfers, thus, the kernel command line must be passed pcie_mem=8M at least. If the memory reserved is less, then the application will exit.

Files

./Makefile
./lib
./lib/Makefile
./lib/ti81xx_ep_lib.c
./lib/ti81xx_mgmt_lib.c
./lib/ti81xx_pci_info.c
./lib/ti81xx_trans.c
./include
./include/debug_msg.h
./include/ti81xx_ep_lib.h
./include/ti81xx_mgmt_lib.h
./include/ti81xx_pci_info.h
./include/ti81xx_trans.h
./application
./application/makefile
./application/ti81xx_ep_app.c
./application/ti81xx_rc_app.c

Building Applications

Create an empty directory and extract the attached tarball epdrv_sample_apps.tar.gz into that directory.

E.g., assuming the directory created is named 'apps' and eprv_sample_apps.tar.gz is copied to it,

# cd apps
# tar -xzvf eprv_sample_apps.tar.gz

Build the applications for TI81XX RC and TI81XX EP using following command

# make KERNSRC=<absolute-path-to-TI81XX-kernel-source>

Remember to pass correct path to kernel source where the kernel tarball mentioned earlier is extracted.

The above command will build EP and RC applications at following paths respectively:

application/ep_app
application/rc_app

In case the RC kernel source is different than EP kernel (e.g., when using PC -> TI81XX setup), the RC and EP side application components need to be built separately in two steps as below example:

# make KERNSRC=<absolute-path-to-RC-kernel-source> rc
# make KERNSRC=<absolute-path-to-TI81XX-EP-kernel-source> ep

You may use make clean in between to force rebuild of the library files which are common for RC & EP. Note that this will also delete the application built in earlier step, so ensure it is copied elsewhere.

To clean the built files, you can use

# make clean

Note: Currently only default mode of applications is supported (DEMO mode). The other 2 modes - throughput and sanity mode are not supported.

Execution

Following sequence needs to be executed to run the demo with above components (commands with 'EP#' to be executed on EP size while those with 'RC#' to be executed on RC side):

• Ensure the desire PCIe setup is done - in this case, we assume TI816X <-> TI816X setup.
• Power on TI816X EP
• Power on TI816X RC
• Boot EP from RC (refer Boot Driver User Guide)

Note: It should be possible to skip this step and instead have EP booted from any non PCIe boot mode (e..g, NAND boot) but it must be ensured that PCIe h/w is initialized, enumerated (from RC) and EP kernel passed memory size to be reserved as mentioned in next step before loading EP driver.

• Make sure that at least 8 MB of PCIe memory is reserved by appending pcie_mem=8M (or more) to the 'bootargs' to the EP kernel. You may also need to ensure that the 'mem' argument passed to kernel ensures sufficient memory is available.
• Ensure that EP Driver, EDMA driver and EP sample application are present in a directory on EP filesystem.
• Once shell is available on EP side, navigate to the directory containing the above files. Load EP Driver and EDMA sample driver:

EP# insmod ti81xx_pcie_epdrv.ko
EP# insmod ti81xx_edma.ko

• After successful load of these modules, run EP application:

EP# ./ep_app

• You may need to add executable permission to the application before above step:

EP# chmod a+x ep_app

• The EP application will not wait for communication with peer.
• On TI816X RC side, make sure that the RC driver and application are present in working directory.
• Navigate to that directory and load the RC driver

RC# insmod ti81xx_pcie_rcdrv.ko

• Run RC application (again, you may need to run chmod a+x rc_app before this step)

RC# ./rc_app

Since these applications are built with DEMO mode, they will start communicating each other and do data transfer in either directions using CPU (read/write from RC) and EDMA (read/write from EP).

During execution of the applications, on RC and EP consoles, you would see chunks with 'C', 'D', 'L' characters. The meaning is explained below:

on RC, data represented by 'C' is sent by memcpy from EP
on RC, data represented by 'D' is sent by EDMA from EP
on EP, data represented by 'C' is sent by memcpy from RC
on EP, data represented by 'L' is data that has been read by EDMA from a buffer on RC populated by application on RC.

• To exit the DEMO, terminate RC and EP apps - this can be done by pressing Ctrl + C combination at shell on both RC and EP. Note that there are no log files generated for DEMO mode.

Getting Throughput from EDMA transfers

For throughput measurements, MSI interrupt from EP to RC are used for control communication.

• You can also build the applications in throughput mode to get the throughput data for EDMA transfers from EP (read/write).
• To set this mode, build the applications by passing MODE=THPT as argument to 'make' command.
• Assuming we are building the applications for TI816X (RC) <-> TI813X (EP), using following commands on build host in EP sample applications directory

# make clean
# make KERNSRC=<path-to-TI81XX-kernel-source>  MODE=THPT

• This will build the applications ep_app and rc_app inside application directory. Transfer them to RC and EP as mentioned earlier and set as executable.

• It is assumed that the required modules are loaded on RC and EP after fresh boot as described earlier to proceed application execution
• On TI813X EP, start ep_app followed by starting rc_app on RC.
• In the example setup, following messages were seen

• On TI813X EP

THPT calculated in TX direction is : 318.012421 MBPS
Executing test case THPT RX (RC read from EP)
THPT calculated in RX direction is : 302.421722 MBPS
Executing test case THPT TX/RX (simultaneous read & write from EP)
THPT calculated in TX direction is : 154.729523 MBPS
THPT calculated in RX direction is : 154.729523 MBPS
Total thpt calculated tx+rx is:  309.459045 MBPS

• On TI816X RC

root@am389x-evm:/home/hemant/hemant/ti816x_pcie# ./rc_app
***Forcing polling (interrupt communication disabled, RC will not generate interrupt)