USB to PCI loopback

Introduction

The purpose of this sample project is to demonstrate how to use the DM6467 to act as a bridge between USB and PCI. This example can be easily extended to bridge USB to another peripheral such as SD/MMC, HDD, or EMAC. The main components of this example are:

• A USB host application that will send data to the DM6467. This application will time how long it takes for the data to come back over USB.
• A User Space USB driver running on the gadgetFS file system on the DM6467 which transfers incoming data to the PCI device.
• A PCI host application that boots the DM6467 over PCI and is responsible for reading and writing data over PCI.

System Design

The design of this system is best shown in the below diagram. The steps are:

1. USB host sends a packet of data to the DM6467
2. DM6467 transfers data to PCI buffer and notifies PCI host that data is available
3. PCI host reads the data over PCI and then writes the data back over PCI to DM6467
4. PCI Host notifies DM6467 that the data is available
5. DM6467 writes the data to the USB Host and the USB Host reports the trip time.

Requirements

1. USB Host device. This can be a PC (although you may notice latency issues if the PC is under load). In this setup a DM6446 EVM was used as the USB Host.
2. DM6467 with a kernel configured for PCI slave and USB peripheral mode. Details on this configuration can be found in the sections below.
3. A PCI Host to boot the DM6467.
4. The DM6467 must be set to boot from PCI. This can be done by setting SW3[1:8] 01000000 where 0=off and 1=on.
5. USB cable to connect the USB Host and DM6467.
6. You may need a PCI extender card like the one pictured below if you cannot find a USB cable that can be used with the EVM while it is plugged into the PCI slot. The extender will allow a normal USB cable to be plugged into the USB connector while the PCI is in use.
7. The USB to PCI loopback sample code. This tarball contains the code for the USB host, DM6467, and PCI Host devices.

• 

  PCI extender

• 

  USB cable connector

USB Host Setup

This section will cover how to configure the USB host applications and drivers for the USB to PCI loopback testing. This section also contains information about the design and usage of the USB host application that moves data to/from the DM6467 over USB.

Kernel Configuration

In order to easily communicate with the GadgetFS driver on the DM6467, generate data packets, and get timing information the usbtest driver is used. On some systems such as RedHat Enterprise Linux 4 this driver is already provided as a loadable module. However, if a DaVinci device is being used as the USB Host (as in this setup) then the usbtest driver must be configured as part of the kernel (as either static or a module). The usb test driver can be found in the kernel configuration under Device Drivers -> USB Support -> USB testing driver (DEVELOPMENT).

Test Application

The USB host test application is named USB_to_PCI_loop_test.c and is located in the USB_Host_App directory of the sample code package. This application is based on the testusb.c application from http://www.linux-usb.org/usbtest/. This application is designed to work with the usbtest driver to send data over the USB connection and get timing data. It has be modified as described in the design section below.

Design

The handle_testdev function has been replaced with a function called run_loopback_test. The run_loopback_test function performs the following operations:

1. Initialize timer structures
2. For number of iterations do
   1. Request usbtest module to write 512 bytes of data.
   2. Record amount of time required for USB write.
   3. Request usbtest module to read 512 bytes of data. This operation will block until the GadgetFS USB driver on the DM6467 has data for the USB Host to read.
   4. Add the amount of time required for the USB read to the time for one loop
3. Report timing statistics

Compilation

This application should be compiled on the USB host. A Makefile is provided so the build procedure is to copy the USB_Host_App directory to the USB Host system and type make in the USB_Host_App directory. This will create a binary executable called USB_to_PCI_loop_test.o

Usage

The USB_to_PCI_loop_test.o application should be invoked using the following command line:

 ./USB_to_PCI_loop_test.o -a -c 1

The -a option specifies to discover all USB devices connected (In this case only the DM6467 should be connected) and the -c option specifies the number of iterations.

 NOTE:  The reason the number of iterations is set to one in the above example is that using more than 1 iteration at a
        time is not currently supported.  It is possible to run the single loop test multiple times but with only 1 
        iteration per run.

DM6467 Setup

This section covers how to setup the DM6467 to accept data from the USB and PCI peripherals.

Kernel Configuration

In order for the DM6467 to communicate over USB and PCI the kernel must be configured for USB GadgetFS and PCI. The steps below cover how to configure the kernel.

1. Create a copy of the LSP kernel. In this example the kernel will be in the /home/user/kernel/ti-davinci directory.
2. Apply this USB GadgetFS highspeed patch to enable the GadgetFS driver in the DaVinci kernel to properly act as a highspeed device. The below instructions assume you have downloaded the patch to your /tmp directory

   cd /home/user/kernel/ti-davinci

   gunzip /tmp/USB_PCI_loopback_gadgetfs_highspeed.patch.gz

   patch -p1 < /tmp/USB_PCI_loopback_gadgetfs_highspeed.patch

3. Configure the kernel with the DM6467 defaults

   make davinci_dm646x_defconfig

4. Open the kernel configuration utility. In this example I am using xconfig.

   make xconfig

5. Enable the DM6467 PCI driver. This driver is located under Device Drivers -> Character devices -> DaVinci PCI Module Support. In this example the driver is configured as a module. You should see a screen similar to the one below.

   

6. Enable GadgetFS support. This requires that you enable Device Drivers -> USB support -> USB Gadget Support -> Support for USB Gadgets AND Device Drivers -> USB support -> USB Gadget Support -> Support for USB Gadgets -> USB Gadget Drivers -> Gadget Filesystem(EXPERIMENTAL)(NEW). In this example these options are build as modules. You should see a screen similar to the one below.

   

7. Configure the DaVinci USB controller for peripheral mode. This requires setting Device Drivers -> USB support -> Inventra USB Highspeed Dual Role Controller Support -> Driver Mode -> USB Peripheral (gadget stack). You should see a screen similar to the one below.

   

8. Compile the kernel uImage and copy it to a tftp directory. In this case the default tftpboot directory is used and the kernel image is named uImage-pcigadgetfs.

   make uImage

   cp arch/arm/boot/uImage /tftpboot/uImage-pcigadgetfs

9. Compile the kernel modules which includes the PCI slave driver and the USB GadgetFS driver.

   make modules

10. Install the modules to your target file system. In this example the root of the file system is an NFS file system located at /home/user/filesys

    make modules_install INSTALL_MOD_PATH=/home/user/filesys

USB User Space Driver

The user space driver being used in this example is based on the usb.c driver from [1]. This driver also requires the usbstring.c and usbstring.h files from the same site. The driver has been modified as described in the Design section below. The sources for this driver can be found in the sample code package in the USB_PCI_Device_App/src directory.

Design

This driver has 2 main threads of execution which are used to accomplish the movement of data between the PCI and USB interfaces. They are described below:

simple_source_thread

This function is a modified version of the original function. This function has been modified to open the PCI device and wait for an interrupt to be triggered indicating that there is data to be written from the PCI buffer over the USB connection. The basic flow of operations is

1. Open the PCI device

   devDesc = open("/dev/dm6467pci". O_RDWR);

2. Loop until program signals threads to exit.
   1. Wait for an interrupt over PCI

      waitForInterrupt(devDesc);

   2. Write the contents of the PCI buffer over USB

      status = write (source_fd, buffer, USB_BUFSIZE);

3. Close the PCI device

   status = close(devDesc);

simple_sink_thread

This function is a modified version of the original function. This function has been modified to open the PCI device and wait data to be written over the USB interface. Once data has been received an interrupt to the PCI host will be triggered indicating that there is data to be read from the PCI buffer. This function is also responsible for mmap'ing the PCI buffer used by both the sink and source threads. The basic flow of operations is

1. Open the PCI device

   devDesc = open("/dev/dm6467pci". O_RDWR);

2. mmap the PCI buffer

   buffer = (char *)mmap(NULL, USB_BUFSIZE, (PROT_READ|PROT_WRITE)

   , MAP_SHARED, devDesc, (off_t)0x11818000);

   NOTE - This function uses the address 0x11818000 for the PCI buffer which is L2 cache memory. If you wish to use a different memory location you will need to change this address here as well as in the PCI host application.

3. Loop until program signals threads to exit.
   1. Read data over USB into PCI buffer. This blocks until data is sent

      status = read (sink_fd, buffer, USB_BUFSIZE);

   2. Interrupt the PCI host to signal that data is available

      sendInterrupt(devDesc);

4. munmap the PCI buffer

   munmap(buffer, USB_BUFSIZE);

5. Close the PCI device

   status = close(devDesc);

Compilation

This driver must be compiled for the DM6467. In the src directory there is a Makefile. You will need to modify the Makefile in order to set the proper KERNDIR value. The KERNDIR variable is the path to the include directory of your LSP tree. In this example KERNDIR is /home/user/kernel/ti-davinci/include If you are compiling on the DM6467 you can simply type make to compile the driver. If you are compiling on a Linux host workstation you will need to specify the CROSS_COMPILE variable in order to use the proper compiler. In most cases you will be cross compiling since you probably will not have your kernel sources installed on the target system.

On the DM6467:

 make

On the Linux host workstation

 make CROSS_COMPILE=arm_v5t_le-

The build will create an executable called usb. This is the user space USB driver and should be copied to the target file system.

Usage

In order to use the usb driver you must setup the GadgetFS file system and driver as well as the PCI slave driver. There is a script in the USB_PCI_Device_App/scripts directory named setup_USB_to_PCI.sh which will do this for you. You should copy this script to your target file system at the same location where the user space USB driver compiled in the previous step was copied. The script can then be executed to setup the file system and start the driver using the command

  ./setup_USB_to_PCI.sh

The steps the script takes are:

1. Create a mount point for the GadgetFS file system

   mkdir /dev/gadget

2. Load the PCI and GadgetFS drivers

   modprobe gadgetfs

   modprobe pcimodule

3. Mount the GadgetFS file system

   mount -t gadgetfs gadgetfs /dev/gadget

4. Start the user space USB driver in verbose mode.

   ./usb -v

After the script has run you should see output on your console similar to:

root@158.218.49.73:~# ./setup_USB_to_PCI.sh
Creating mount point
Loading Drivers
gadgetfs: USB Gadget filesystem, version 24 Aug 2004
pcimodule: module license 'unspecified' taints kernel.
Mounting USB Gadget File System
Starting USB Gadget Driver
DEVNAME = musb_hdrc
gadgetfs: bound to musb_hdrc driver
/dev/gadget/musb_hdrc ep0 configured
serial="o2sse8g556z467tf41bt9n0ltyou48gaew0rwnvmc10h9blw3x86qqg1crkmjwf"
gadgetfs: suspended from state 2
SUSPEND

Once the DM6467 is connected via USB to a device running the usbtest module you should see output similar to:

gadgetfs: connected
CONNECT high speed
SETUP 80.06 v0300 i0000 255
SETUP 80.06 v0302 i0409 255
SETUP 80.06 v0301 i0409 255
SETUP 80.06 v0303 i0409 255
SETUP 80.06 v0303 i0409 255
SETUP 80.06 v0303 i0409 255
SETUP 80.06 v0303 i0409 2
SETUP 80.06 v0303 i0409 128
gadgetfs: configuration #3
SETUP 00.09 v0003 i0000 0
CONFIG #3
simple_source_thread start 1083468640 fd 5
simple_sink_thread start 1091857248 fd 7

The last two lines indicate that the source and sink threads are running and waiting for data. You can exit this user space USB driver at any time by pressing CTRL + C. You will see output similar to:

ep0 gadgetfs: disconnected
read after poll: Interrupted system call
done

PCI Slave Driver

The PCI slave driver is included as part of the LSP kernel and can be used without modification.

Design

For more information on the design of the PCI slave driver please see the LSP 1.30 DM6467 PCI Slave Driver User's Guide document in the docs/dm646x directory of the DVSDK PSP directory.

Usage

This driver is loaded by using the modprobe pcimodule command. It creates the device node /dev/dm6467pci which can be used to interract with the driver.

PCI Host Setup

This section will cover the steps required to use the PCI host driver and application for the data loopback as well as booting of the DM6467.

PCI Host Driver

This driver is based on the PCI host driver provided as part of the PSP package. This driver is used to do the I/O operations over the PCI bus to/from the DM6467. The original driver sources can be found in the PSP/board_utilities/dm646x/pci_boot.tar.gz package in the DVSDK. The driver used in this demo has been modified as described in the design section below and can be found in the PCI_Host_App/src directory in the sample code package.

Design

This driver required modification to allow a user space application to block while waiting for an interrupt. This was done by using a simple ioctl operation that was added to the driver. The new ioctl is named DM646X_PCI_WAIT_INTRPT and takes no arguments. This ioctl performs a wait_for_completion operation and the interrupt service routine of the driver has been modified such that if there is a non-NULL wait structure then a completion is performed. In this case the user space application that performs the data read/write operations can wait for data to become available from the DM6467.

For more details on the rest of the driver functionality please see the LSP 1.30 DM6467 PCI Boot Driver User's Guide in the PSP/docs/dm646x directory of the DVSDK.

Compilation

The Makefile in the src directory is setup to compile the driver for the running kernel as well as the Host application that will boot the board and allow the user to perform a PCI loopback. To build the driver and host application on the PCI Host system just type make in the src directory. This will create a driver module dm646x_pci_boot.ko and the user space application executable pci_loopback_host_app.o.

NOTE: This driver has been validated on RedHat Enterprise Linux 4. It may require some modification for other distributions.

Usage

This driver requires no parameters and just needs to be loaded with the insmod command on the PCI host system. This driver will need to be loaded each time the PCI host machine is rebooted. NOTE: You will need to execute the insmod command with root permissions.

insmod ./dm646x_pci_boot.ko

You should see output like the following in the messages log when the driver is loaded.

dm646x_pci_boot: module license 'unspecified' taints kernel.
DM646x PCI Boot: Major number 254 assigned
PCIDRV - Added device to the sys file system
Found DM646x PCI device at 0xc1670c00
PCIDRV - Found PCI Devices
PCIDRV - BAR Configuration -
          Start        |       Length  |       Flags
       0xff8f8000      |       32768   |       0x00000200
       0xff8f0000      |       32768   |       0x00000200
       0xff400000      |       4194304 |       0x00000200
       0xf6ae0000      |       131072  |       0x00001208
ARM TCM RAM memory mapped to 0xe0b40000
DDR2 Memory Controller registers mapped to 0xe0b50000
MMR registers mapped to 0xe0b80000
Original BAR-4 value    = 0x80000000
New BAR-4 value =       0x82000000
Original BAR-5 value    = 0x80800000
New BAR-5 value =       0x82800000
       0xf6000000      |       8388608 |       0x00001208
       0xf5800000      |       8388608 |       0x00001208
DDR2 memory mapped to 0xe1000000
Communication/Script area mapped to 0xe1100000
Linux area mapped to 0xe1200000
DDR2-B memory mapped to 0xe1980000
ACPI: PCI Interrupt 0000:01:0c.0[A] -> GSI 17 (level, low) -> IRQ 185
PCIDRV - Device enabled
PCIDRV - Driver Data Set
PCIDRV - Set as the Bus-Master
Interrupt number is : 185
PCIDRV - Interrupts registered

UBL completed its job; BC is cleared; DDR2 is initialized
Original BAR-5 value    = 0x82800000
New BAR-5 value =       0x83000000
Original BAR-5 value    = 0x83000000
New BAR-5 value =       0x82800000
Original BAR-4 value    = 0x82000000
New BAR-4 value =       0x81000000
Original BAR-4 value    = 0x81000000
New BAR-4 value =       0x82000000

PCI Host Application

The PCI host application is based on the saBootApp application from the PSP. This application is responsible for booting the DM6467 device over the PCI bus and then performing the PCI loopback functionality.

Design

The primary modification made to the saBootApp application was to add the loopback command to the console_control function. This command will cause the application to go into an infinite loop in which the following steps are performed:

1. Begin Loop
2. Wait for an interrupt from the DM6467

   result = ioctl(dev_desc, DM646X_PCI_WAIT_INTRPT);

3. Read the data over PCI and then write it back over PCI

   result = loop_dma_once();

4. Send an interrupt to the DM6467 indicating data is available

   result = ioctl(dev_desc, DM646X_PCI_GEN_INTRPT);

NOTE: This application uses the address 0x11818000 which matches the address used by the user space USB driver on the DM6467. If you wish to use another address you will need to modify the location variable in the loop_dma_once function to the new address.

Usage

The steps to build this application were covered in the compilation section of the PCI host driver above. To use this application the following steps need to be performed.

• Create a working directory on the PCI host machine and copy the PCI_Host_App directory to it. For this example the working directory will be /home/user/workdir

  mkdir /home/user/workdir

  cp -rf <sample code location>/PCI_Host_App /home/user/workdir

  cd /home/user/workdir/PCI_Host_App

• Create a directory to place the executables in.

  mkdir bin

• Copy the UBL to your working directory. The UBL for PCI boot mode can be found in the PSP/bin/dm646x directory of the DVSDK and is called ubl_dm646x_297_pci.bin. The UBL must be named ublDaVinci.bin and needs to be in the same directory as the PCI host application.

  cp <PSP dir>/bin/dm646x/ubl_dm646x_297_pci.bin bin/ublDaVinci.bin

• Copy the U-Boot bootloader for PCI to the bin directory. The U-Boot for PCI boot mode can be found in the PSP/bin/dm646x directory of the DVSDK and is called u-boot_pci.bin.

  cp <PSP dir>/bin/dm646x/u-boot_pci.bin ./bin/

• Copy the target kernel image created in the Kernel Configuration section to the bin directory. NOTE: If you are booting over tftp you still need the kernel image in the bin directory to be copied to the DM6467 over PCI, however it will not be used. If you are not using tftp this is the kernel image that will be booted on the board. User's may find it useful to use tftp during development and then switch to using the uImage over PCI when the kernel image is solidified.

  cp <target kernel image> ./bin/

• Copy the target file system image to the bin directory. NOTE: If you are booting using an NFS root file system you will still need to provide a file for the ramdisk, but it can be any file and does not have to be a valid ramdisk file system. If you are not booting using NFS then you will need to provide a valid ramdisk file system. User's may find it useful to use tftp during development and then copy their application files to a ramdisk to be used over PCI when the application is done with development.

  cp <ramdisk> ./bin/

• Create a boot script image. Examples can be found in the PCI_Host_App/scripts directory. The bootnetwork.txt script is an example for booting the board over tftp using an NFS root file system. the bootscript.txt is an example of how to boot the board over PCI in a stand alone mode without requiring networking. For more information see the README.txt file in the scripts directory. After creating the script file you must create a binary image of the script to be sent to the DM6467. This can be done by using the following command:

  mkimage -T script -A arm -C none -n 'Boot Script File' -d <script name> <output name>

  i.e. mkimage -T script -A arm -C none -n 'Boot Script File' -d bootnetwork.txt bootnetwork.img

• Copy the boot script image to the bin directory.

  cp bootnetwork.img ./bin/

• Copy the PCI host driver to the bin directory

  cp src/dm646x_pci_boot.ko ./bin/

• Copy the PCI host boot application to the bin directory

  cp src/pci_loopback_host_app.o ./bin/

• Copy the helper script run_test.sh from the PCI_Host_App directory to the bin directory.

  cp run_test.sh ./bin/

• Edit the run_test.sh script and change the variables PCI_DRIVER, UBOOT, UIMAGE, RAMDISK, and BOOTIMG to reflect your setup.
• Run the run_test.sh script. This script will insmod the PCI host driver and start the PCI host application. Once the application starts you should see output similar to:

Loading PCI Host driver
Executing Boot and Host Application
ublDaVinci.bin file opened
Size of ublDaVinci.bin file = 14336
14336 bytes read from file ublDaVinci.bin
networkboot.img file opened
Size of networkboot.img file = 496
496 bytes read from file networkboot.img
Bar-5 mapped to 0xb7785000
ramdisk.gz file opened
Size of ramdisk.gz file = 2393160
1048576 bytes read from file ramdisk.gz
Bar-5 remapped to 0x83000000
1344584 bytes read from file ramdisk.gz
Bar-5 remapped to 0x82800000
uImage file opened
Size of uImage file = 1379004
1379004 bytes read from file uImage
u-boot_pci.bin file opened
Size of u-boot_pci.bin file = 99100
99100 bytes read from file u-boot_pci.bin

Type "help" or "?" for available commands
PCIConsole:\>

• At the PCIConsole:> prompt type loopback and press ENTER. This will start the PCI host application to begin waiting for data to loopback over the PCI bus.

Putting it all Together

Now that you know how to build and run each of the three components of this sample application it is time to cover how to use them all together. In order to get the timing information for a 512 byte packet to be transfered over USB, to PCI, back over PCI, and then back over USB the following steps need to be performed.

Test Setup:

• Kernel Image: uImage-pcigadgetfs
• Root File System: NFS file system located at /home/user/filesys
• Boot Method: PCI boot with kernel over tftp and NFS root file system
• Connections:
  • USB cable connected between DM6467 and USB host
  • Network cable connected to DM6467
  • Serial cable connected to DM6467

Test Procedure:

1. Boot the USB host system. In this example the host system is a DM6446 EVM running MontaVista Linux.
2. Boot the PCI host system. In this example the PCI host system is a Linux workstation running RedHat Enterprise Linux 4.0 with the DM6467 plugged in to one of the PCI slots.
3. Open a serial terminal to view the boot messages of the DM6467 generated in the next step.
4. Boot the DM6467 using the PCI host application. Refer to the Usage section of the PCI host application above for how to boot the DM6467 board.
5. When the DM6467 has finished booting login as root
6. Start the user space USB driver as detailed in the usage section of the USB user space driver.
7. On the PCI host type loopback at the PCIConsole:\> prompt
8. On the USB host run the USB_to_PCI_loop_test.o application as detailed in the usage section of the USB host test application section.
9. On the USB host you should see output similar to:

unknown speed   /proc/bus/usb/001/002
loop took    0.001188 sec


Maximum Loop Time =     0.001188 sec
Minimum Loop Time =     0.001188 sec
Average Loop Time =     0.001188 sec
Total Loop Time   =     0.001188 sec

You can rerun this test continously if you wish. The average time observed is ~1ms to transfer 512 bytes of data over USB and PCI.