Enabling Multiple Ethernet Ports on TI81xx Devices for Substation Automation Applications

Introduction

The revamping of power grids across the world requires smart devices with the capability to monitor multiple networks. These devices are used within electrical substations where the ability to handle multiple communication links between equipment and the central office is a key requirement. These systems require application processors with multiple Ethernet connections for fault-tolerance purposes.

The DM816x/C6A816x class of application processors include two Gigabit Ethernet controllers. However, substation automation applications require more than two Ethernet ports.

This article describes how a a PCIe-to-PCI bridge can be used to expand the number of Ethernet ports on the C6A816x EVM. The two Gigabit Ethernet ports on the C6A816x device operate in parallel with three PCI 10/100 network interface cards connected through the bridge.

Performance benchmarks are also presented to give an idea of the bandwidth achieved through each Ethernet port as well as the loading on the C6A816x ARM core.

NOTE: The hardware and sotware setup instructions on this wiki are specific to DM816x/C6A816x devices.  However, they can be extended to DM814x/C6A814x devices since they share a common hardware architecture and software base.  For the purposes of this wiki, the term "TI81xx" encompasses both DM816x/C6A816x and DM814x/C6A814x devices.

MultiEthernet HW Setup.PNG

MultiEthernet HW.PNG

Prerequisites

This guide assumes that the development environment is setup, and that UBL and U-boot has been flashed on the C6A816x EVM. If not, please refer to the documentation that is part of the PSP release for more information.

Hardware

• C6A816x EVM
• XIO2001 (x1 PCI Express to PCI Bus Translation Bridge) EVM
• Network Interface Cards (NIC) with RealTek 8139 chipset
  • Note: This system has only been validated with the RealTek 8139 chipset using the 8139too Linux Ethernet driver
  • Intellinet 10/100 Ethernet LAN PCI Card
  • D-Link 10/100 Desktop PCI Adapter (rev d2)

Development Environment

• Connect to the serial port, power on the board, and verify that the UBL and U-boot prints from serial port terminal
• Verify that the C6A816x EZSDK is installed, and that the kernel source code is version 2.6.37 or higher. If it is not, go to the EZSDK software page and download the latest software.
• Verify that the TFTP server is configured in your host Linux machine
  

Configure Kernel options and modify drivers

• Setup environment variables

$ source <EZSDK directory>/linux-devkit/environment-setup
$ cd <EZSDK directory>/psp/linux-#.#.#.#

• Setup default configuration by executing the following command

$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- ti8168_evm_defconfig

• Modify the kernel by entering into the interactive configuration with the following command

$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- menuconfig

• Configure kernel to enable Realtek 8139 driver

1. Highlight "Device Drivers" and hit Enter
2. Highlight "Network device support" and hit Enter
3. Highlight "Ethernet (10 or 100Mbit)", hit "Y", and hit Enter
4. Highlight "EISA, VLB, PCI, and on board controllers" and hit "Y"
5. Highlight "RealTek RTL-8129/8130/8139 PCI Fast Ethernet Adapter support" and hit "M"
6. Highlight "Use PIO instead of MMIO" and hit "N"
7. Save configuration

• Since the PCIe peripheral maximum remote read request size is 256 bytes, the following will modify the PCI driver to restrict XIO2001 read request size to 256 bytes. By default, the read request size of XIO2001 is 512 bytes.

$ cd <EZSDK directory>/psp/linux-#.#.#.#
$ sudo gedit drivers/pci/quirks.c

At the bottom of the file, add the following:

static void __devinit quirk_ti_xio2001_read_req_size(struct pci_dev *dev)
{ WARN_ON(pcie_set_readrq(dev, 256) < 0); }

DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_XIO2001, quirk_ti_xio2001_read_req_size);

Save quirks.c

$ sudo gedit include/linux/pci_ids.h

Add the following under line 786:

#define PCI_DEVICE_ID_TI_XIO2001	0x8240

Save pci_ids.h

• Recompile kernel

$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- uImage

• Recompile modules

$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- modules

• Copy compiled binaries to the TFTP folder

$ cp arch/arm/boot/uImage /tftpboot
$ cp drivers/net/*.ko /tftpboot

Load Kernel and Drivers

• Flash the kernel to the desired boot mode by following the instructions found in the U-Boot User Guide, which is available in the PSP release.

• Transfer the driver modules to the running C6A816x file system. You can use TFTP to get the files on the C6A816x EVM as follows. Note that in the example, we assume TFTP server 172.24.190.43 is running with the concerned files copied into the server's TFTP folder.

TI8168_EVM# cd /home/root
TI8168_EVM# tftp -g -r mii.ko 172.24.190.43
TI8168_EVM# tftp -g -r 8139too.ko 172.24.190.43

• Note that the above commands may vary depending upon the filesystem and the tftp utility being used. Also, you can use different means of transferring the above files into the C6A816x filesystem as convenient. Assuming the files are transferred successfully, and you have navigated to the directory containing the above files, use the following commands to load the network drivers.

TI8168_EVM# cd /home/root
TI8168_EVM# ls              // Verify that you see 8139too.ko and mii.ko

• Execute the following commands:

TI8168_EVM# insmod mii.ko
TI8168_EVM# insmod 8139too.ko

• If everything went well, you should be able to see all Ethernet ports now.

TI8168_EVM# ifconfig -a

Performance and Benchmarks

The following benchmarks were collected on a Multiple Ethernet setup that has three 10/100 NIC cards and two Gigabit Ethernet ports.  Each Ethernet port directly connects to a Host PC with a crossover cable.  In the test environment, a total of 5 Host PCs were used. 

Benchmark

Network Configuration	CPU Loading (ARM Only)	Throughput
1 10/100 NIC(1)	ARM Load = 18%	NIC_1 = 94.9 Mbits/s
2 10/100 NIC(1)	ARM Load = 32%	NIC_1 = 44.8 Mbits/s NIC_2 = 57.6 Mbits/s
3 10/100 NIC(1)	ARM Load = 51%	NIC_1 = 41.9 Mbits/s NIC_2 = 38.6 Mbits/s NIC_3 = 38.8 Mbits/s
1 GigE	ARM Load = 95%	GigE_1 = 591 Mbits/sec
2 GigE	ARM Load = 99%	GigE_1 = 273 Mbits/sec GigE_2 = 275 Mbits/sec
1 GigE 2 10/100 NIC	ARM Load = 99%	GigE_1 = 196 Mbits/s NIC_1 = 32.6 Mbits/s NIC_2 = 32.5 Mbits/s NIC_3 = 34.7 Mbits/s
2 GigE 3 10/100 NIC	ARM Load = 99%	GigE_1 = 117 Mbits/s GigE_2 = 121 Mbits/s NIC_1 = 29.1 Mbits/s NIC_2 = 30.4 Mbits/s NIC_3 = 28.8 Mbits/s

(1) In cases where the ARM load is less than 99-100%, the window size can be tweaked to get higher performance.  

Notes

• All ARM load numbers are approximations/averages of numbers read using top command on Linux.
• The performance numbers were captured using the iperf v2.0.4 tool in unidirectional mode. Usage details are mentioned below:
• On the C6A816x DUT iperf is invoked in server mode with a window size of 212K.  NOTE: Linux default window size of 212K.
• On PC Hosts invoked iperf in the client mode with a window size of 128K (unless otherwise noted).  
• The transfers are measured over a duration of 60Secs
• Cross cable is used to measure performance
• C6A816x are running at an ARM speed of 987MHz (unless otherwise noted) and DDR2 speed of 398MHz.
  
• For all testing, root filesystem mounted from SD card.
• Gigabit ports speed is set to 1000Mbps; NIC speed is set to 100Mbps.