TI81XX PCIe FAQs

TI81XX PCI Express FAQs

Hemant Pedanekar

This page is currently under construction. The content of this page is due to change quite frequently and thus the quality and accuracy are not guaranteed until this message has been removed. Please feel free to contribute to this page while construction is in progress.

Introduction

This document is applicable to DM816x/AM389x, DM814x/AM387x and DM813x family of devices referred hereafter as TI816X, TI814X and TI813X respectively.

Descriptions common across these device families use the term TI81XX/ti81xx.

This document contains answers for Frequently Asked Questions (FAQs) related to PCI Express module on TI81XX devices.

The FAQs are organized in two categories: Hardware specific and Software specific with further distinction related to mode of operation - Root Complex (RC) or Endpoint (EP). Some board (EVM) specific FAQs are also covered at the end.

It is suggested to go through TI816x PSP user document for RC Driver and EP Boot Driver for more details and clarification of PCIe terms used in this document.

Hardware Related FAQs

Common Across RC and EP Modes

Is splitting x2 lanes on TI816x PCIe port into 2 separate ports having 1 lane each possible?

No. The PCIe hardware on TI816x does not support this feature and can only act as a 2-lane, single port PCI Express device. In other words, PCIe link bifurcation is not supported by the hardware.

I have connected TI81xx to a GEN2 capable device and link is established fine but why does the Link Speed field in Link Status register still show GEN1 (2.5 GT/s)?

TI81xx devices do not support h/w autonomous link up-training to GEN2 form GEN1 in either EP/RC modes and if the peer device (RC/EP/Switch) has the same restriction (e.g., when connected to other TI81xx device), then the software needs to initiate speed change. This can be done by setting the "Directed Speed Change" bit (DIR_SPD) in PL_GEN2 register (address @0x5100180c, bit-17) to '1' after the link is established (L0 state).

Setting this bit should re-trigger link training to terminate to L0 state with new speed (GEN2) on success.

Note that this bit can be set on TI81xx device irrespective of its mode of operation - RC or EP - and setting this bit on any one of the link peers should be sufficient.

E.g., when connecting TI81xx RC to TI81xx EP, you can set the PL_GEN2 bit-17 on either of the RC or EP to initiate the speed change to GEN2.

Note: Current implementation of TI816x RC driver (as of 04.00.00.10) does not set this bit. You can either modify the driver or set PL_GEN2 from shell (using /dev/mem interface or devmem2 utility compiled from source available here

Do TI81XX devices support multicast in RC/EP modes?

No. Multicast capability is not supported as TI81XX devices comply to 2.0 and multicast was introduced in 2.1 PCIe spec.

A system involving TI81XX Endpoints may still use multicast capable switch to utilize multicast at switch level for writing to to multiple Endpoints.

How can I force x1 link on TI816x device when connected to an x2 (and up) capable peer?

TI816x devices are x2 link devices (2 lanes) and when connected to a peer with link capability of x2 and onwards (e.g., x2, x4, x8 or x16) will negotiate to x2. If you want to force x1 operation in this case, then this can be achieved by setting following bits in corresponding registers before enabling link training (LTSSM).

 LINK_CAP[9:4] = 1 i.e., MAX_LINK_WIDTH = 1 @0x5100107c
 PL_LINK_CTRL[21:16] = 1 i.e., LNK_MODE = 1 @0x51001710

Note that alternatively you may be able to force x1 link from the peer, if it has support for such configuration.

Also, this configuration can be set even after link is already established in x2 mode. In such case, the link needs to be disabled before setting above configuration and then enable and trigger link training.

TBD

RC Mode Operation

TBD

EP Mode Operation

Can the host (Root Complex) access peripherals such as Timer, UART etc on TI816x EP?

Direct access from peer to L4 peripherals on TI816x device is not possible as there is not connectivity from PCIe master to L4 peripheral space. Please refer the connectivity section in TRM for more details.

How can I generate Legacy Interrupt (INTx) from DM81x EP device?

• Ensure that the MSI capability of DM81x device is disabled by setting bit 16 in MSI Capability register @0x51001150 (when accessing from DM81x EP locally) or @0x51002150 (when accessing from RC) is cleared.
  • Note that this is the default setting when using PCIe boot mode on the DM81x EP
• A (software) write of value 1 to EP_IRQ_SET @0x51000064 on local DM81x EP will generate (assert) the INTA (legacy) interrupt to the RC
• Depending upon the implementation - which would decide (software or hardware) condition on DM81x EP to mark de-assertion of interrupt - the software on EP will need to write 1 to EP_IRQ_CLR @0x51000068 to indicate the RC about interrupt condition being cleared (de-asserted).

Software Related FAQs

RC Driver

I have connected a PCIe to SATA card in PCIe slot on EVM but it is not getting detected

Check the 'lspci' output or check the 'type' entry for all non RC devices inside /sys/bus/pci/devices directory and see if there is a intermediate bridge device being detected (e.g., PCIe-PCI bridge). If yes, then this is most likely related to the issue detected on TI816x releases prior to 04.00.00.07 which was related to "Intermediate bridge devices not getting detected". Use kernel from PSP release 04.00.00.07 or later which has workaround for this issue.

For more information, refer to the mail thread associated to this post: http://www.spinics.net/lists/linux-pci/msg09740.html

How can I enable debugging the PCI/PCIe enumeration and configuration?

Assuming the kernel is already configured for your system (TI816x or TI814x) and build setup is available, execute following at shell from kernel source directory:

make menuconfig

This will present kernel configuration menu. navigate and select "PCI Debugging" option as shown below by pressing SPACE/ENTER key:

--> Bus support --> PCI Debugging

Build the kernel. Make sure to append "debug" to the bootargs to see the verbose messages during boot (you may also want to turn on low level debugging).

How can I set up BAR sizes?

The sizes for windows targeting BAR1 and onwards can be set by writing size mask values to BAR mask registers. BAR mask registers are nothing by overlaid registers on corresponding BAR registers accessible from local master, i.e., Cortex-A8 on local TI81xx device. To access BAR mask registers, the DBI_CS2 bit (bit-5) in CMD_STATUS register (@0x51000004) must be set to '1'.

Following steps explain the operation for setting BAR1 for 1MB size:

1. Set DBI_CS2=1 by setting bit-5 of CMD_STATUS
2. Readback CMD_STATUS to ensure the write went through to avoid conflicts due to clock domain differences of DBI_CS2 bit (Direct Bus Interface).
3. Enable the BAR by writing '1/ to BAR1 register (@0x51000014).
4. Set the mask for desired size. In our example, write 0xFFFFF to BAR1 register.
5. Clear DBI_CS2 bit

How can I initialize PCIe module on DM816x to be used as PCIe Endpoint without using any of the PCIe Boot mode?

When DM816x device is set up in PCIe boot mode (32-bit or 64-bit), the ROM code takes care of initializing the PCIe module so that the upstream device (E.g., Root complex) can establish the link with DM816x EP. If you intend to use a non-PCIe boot mode, this initialization can be done from the boot loader on EP. Assuming NAND boot mode is used and U-Boot is booted, you can use following command sequence at U-Boot prompt:

• Setup SerDes PHY PLL for 2.5GHz by writing to PCIe CFG register in Control Module @ 0x48140640

TI8168_EVM# mw 0x48140640 0x01C90000 

• Note that the above setting also ensure that the PCIE_DEVTYPE field in the above register is set as '0' (LS 2 bits) which means EP mode of operation.
• Wait for bit 8 in the above register to become '1' to indicate lock status. You can use 'md 0x48140640' command at U-Boot prompt to dump the contents.
• Enable the clockdomain associated with PCIe module

TI8168_EVM# mw 0x48180510 2

• Enable PCIe functional clock

TI8168_EVM# mw 0x48180578 2

• De-assert to local reset to PCIe module by clearing bit 7 of PRCM register RM_RSTCTRL @ 0x48180B10. Assuming the value read form this register was 0xff, write following to de-assert LRST

TI8168_EVM# mw 0x48180B10 0x7F

• Initiate Link training (LTSSM) by setting bit 0 of PCIe CMD_STATUS register in application registers space @ 0x51000004. Assuming the reset value is 0xa00, use following to enable LTSSM

TI8168_EVM# mw 0x51000004 0xa01

• Power up the RC

Note: If U-Boot (or any bootloader) is not available or cannot be used on EP, the above steps can be performed by using a JTAG connection to Cortex A8 on the DM816x EP (e.g., Using CCSv4 and a GEL file having above steps).

How can I initialize PCIe module on DM816x to be used as PCIe Root Complex without using Linux Kernel integrated with DM816x Root Complex Driver ?

The initialization for setting up Root Complex can be done either from boot loader on the device to be used as RC. In below example, we assume NAND boot mode is used and U-Boot is booted. Following command sequence at U-Boot prompt will set up DM816x as PCIe RC:

• Power up the RC
• Setup SerDes PHY PLL for 2.5GHz by writing to PCIe CFG register in Control Module @ 0x48140640

TI8168_EVM# mw 0x48140640 0x01C90002 

• Note that the above setting also ensure that the PCIE_DEVTYPE field in the above register is set as '2' (LS 2 bits) which means RC mode of operation.
• Wait for bit 8 in the above register to become '1' to indicate lock status. You can use 'md 0x48140640' command at U-Boot prompt to dump the contents.
• Enable the clockdomain associated with PCIe module

TI8168_EVM# mw 0x48180510 2

• Enable PCIe functional clock

TI8168_EVM# mw 0x48180578 2

• De-assert to local reset to PCIe module by clearing bit 7 of PRCM register RM_RSTCTRL @ 0x48180B10. Assuming the value read form this register was 0xff, write following to de-assert LRST

TI8168_EVM# mw 0x48180B10 0x7F

• Initiate Link training (LTSSM) by setting bit 0 of PCIe CMD_STATUS register in application registers space @ 0x51000004. Assuming the reset value is 0xa00, use following to enable LTSSM

TI8168_EVM# mw 0x51000004 0xa01

• Power up the any endpoint (EP) device(s) in the system (Note: This can also be done with RC power up itself).
• You will still need software to find these EP devices and allocate resources to them before accessing.

Note: If U-Boot (or any bootloader) is not available or cannot be used on RC, the above steps can be performed by using a JTAG connection to Cortex A8 on the DM816x EP (e.g., Using CCSv4 and a GEL file having above steps).

EP Boot Support

what’s the difference between pcie boot and SPI pcie boot support?

1. PCIe Boot mode: Here ROM runs on EP and helps RC s/w to load U-Boot stage which performs DDR etc init
2. PCIe boot using U-Boot: Here U-Boot is in EP side SPI flash and should already have DDR init code for particular h/w (board). It also carries out laoding and running of kernel over PCIe from RC side s/w (no need of 2nd stage U-Boot).

• The 2nd case allows lot of customizations / fixes compared to using ROM boot methods –
  • E.g., setting different BAR sizes than possible using pins as in case of ROM boot
  • Bug fixes such as putting valid class code, device id etc
• It also enables recovery in case the RC issues PCIe reset, link status change during boot.
  • This particularly fixes one of the major issues when DM81xx devices could be unusable over PCIe when connected to certain PCs/RCs

EP Boot Driver

TBD

EP Driver Package

TBD

EVM/Board Specific FAQs

I have connected DM816x EVM's PCIe slot to a PC's PCIe slot using a male-male cable but why doesn't the BIOS detect the EVM?

Make sure you have set up the DM816x device to boot in PCIe boot mode. This will ensure that the ROM code on DM816x device would initialize the PCIe interface so that the PC detects the device during PCI enumeration. The PCIe boot mode is enabled by setting up DIP switches on the EVM.

The PCIe Boot Driver User Guide (PDF) from PSP release folder has the details about switch settings. The same document is externally available on Wiki at http://processors.wiki.ti.com/index.php/TI81XX_PCI_Express_Endpoint_Boot_Driver_User_Guide.

In addition, the EVM and cable may need modifications to be able to work with PC. Refer app note: http://processors.wiki.ti.com/index.php/DM816x_AM389x_PCIe_Clocking_Schemes