ferdigy at gmail
Nov 5, 2009, 11:33 AM
Post #1 of 1
(1615 views)
Permalink
|
|
Enabling DCA on Dell PowerEdge R710
|
|
Just to share my experience in enabling DCA on a Dell PowerEdge R710 on
CentOS 5.4 32-bit (2.6.30-9) using this method. Thanks to Joe
Damato's dca_force.
http://timetobleed.com/enabling-bios-options-on-a-live-server-with-no-rebooting/
Pre-requisites:
1. Check if DCA is disabled on your Dell PowerEdge server: modprobe ioatdma;
dmesg | grep ioatdma
ioatdma 0000:00:16.0: PCI INT A -> GSI 51 (level, low) -> IRQ 51
ioatdma 0000:00:16.0: setting latency timer to 64
ioatdma 0000:00:16.0: Intel(R) I/OAT DMA Engine found, 1 channels, device
version 0x30, driver version 3.64
ioatdma 0000:00:16.0: irq 73 for MSI/MSI-X
ioatdma 0000:00:16.0: DCA is disabled in BIOS
ioatdma 0000:00:16.1: PCI INT B -> GSI 52 (level, low) -> IRQ 52
ioatdma 0000:00:16.1: setting latency timer to 64
ioatdma 0000:00:16.1: Intel(R) I/OAT DMA Engine found, 1 channels, device
version 0x30, driver version 3.64
ioatdma 0000:00:16.1: irq 74 for MSI/MSI-X
ioatdma 0000:00:16.1: DCA is disabled in BIOS
ioatdma 0000:00:16.2: PCI INT C -> GSI 53 (level, low) -> IRQ 53
ioatdma 0000:00:16.2: setting latency timer to 64
ioatdma 0000:00:16.2: Intel(R) I/OAT DMA Engine found, 1 channels, device
version 0x30, driver version 3.64
ioatdma 0000:00:16.2: irq 75 for MSI/MSI-X
ioatdma 0000:00:16.2: DCA is disabled in BIOS
ioatdma 0000:00:16.3: PCI INT D -> GSI 54 (level, low) -> IRQ 54
ioatdma 0000:00:16.3: setting latency timer to 64
ioatdma 0000:00:16.3: Intel(R) I/OAT DMA Engine found, 1 channels, device
version 0x30, driver version 3.64
ioatdma 0000:00:16.3: irq 76 for MSI/MSI-X
ioatdma 0000:00:16.3: DCA is disabled in BIOS
2. Make sure that I/OAT is enabled in the R710 BIOS menu: "Integrated
Device" -> "I/OAT DMA Engine" -> "Enabled". This is Disabled by default.
3. Removed pciutils and pciutils-devel RPMs then I build and install
pciutils-3.1.4.tar.gz (make clean;make SHARED=yes;make install-lib;make
install)
4. Make sure that these modules are in your running kernel:
CONFIG_X86_MSR=m
CONFIG_INTEL_IOATDMA=m
CONFIG_DMA_ENGINE=y
CONFIG_NET_DMA=y
CONFIG_DCA=m
CONFIG_IGB=m
CONFIG_IGB_DCA=y
Note that I'm using Intel 82575GB NIC (igb driver).
Downloaded and modified dca_force.c because R710 uses Intel 5520 chipset
(see the page 440 of http://www.intel.com/Assets/PDF/datasheet/321328.pdf to
)
-snip-
#define INTEL_BRIDGE_DCAEN_OFFSET 0x8c
#define INTEL_BRIDGE_DCAEN_BIT 0
-snip-
Note: I think I got the right registers but let me know if I made a mistake.
Compiled it with:
cc -Wall -DNUM_CPUS=4 -o dca_force dca_force.c -lpci -lz
Tested it:
modprobe msr;sleep 3;./dca_force;sleep 3;modprobe ioatdma
dmesg
You should see something like this:
ioatdma 0000:00:16.0: PCI INT A -> GSI 51 (level, low) -> IRQ 51
ioatdma 0000:00:16.0: setting latency timer to 64
ioatdma 0000:00:16.0: Intel(R) I/OAT DMA Engine found, 1 channels, device
version 0x30, driver version 3.64
ioatdma 0000:00:16.0: irq 73 for MSI/MSI-X
igb: eth0: __igb_notify_dca: DCA enabled
igb: eth1: __igb_notify_dca: DCA enabled
ioatdma 0000:00:16.1: PCI INT B -> GSI 52 (level, low) -> IRQ 52
ioatdma 0000:00:16.1: setting latency timer to 64
ioatdma 0000:00:16.1: Intel(R) I/OAT DMA Engine found, 1 channels, device
version 0x30, driver version 3.64
ioatdma 0000:00:16.1: irq 74 for MSI/MSI-X
igb: eth2: __igb_notify_dca: DCA enabled
igb: eth3: __igb_notify_dca: DCA enabled
ioatdma 0000:00:16.2: PCI INT C -> GSI 53 (level, low) -> IRQ 53
ioatdma 0000:00:16.2: setting latency timer to 64
ioatdma 0000:00:16.2: Intel(R) I/OAT DMA Engine found, 1 channels, device
version 0x30, driver version 3.64
ioatdma 0000:00:16.2: irq 75 for MSI/MSI-X
ioatdma 0000:00:16.3: PCI INT D -> GSI 54 (level, low) -> IRQ 54
ioatdma 0000:00:16.3: setting latency timer to 64
If it works, put "modprobe msr;sleep 3;/path/to/dca_force;sleep 3;modprobe
ioatdma" on your /etc/rc.d/rc.local.
Hope this helps.
Also, don't forget to use PF_RING-aware igb drivers found in
PF_RING/drivers/igb-2.0.6/src/ (make clean;make CFLAGS_EXTRA="-DIGB_DCA
-I/usr/src/linux-2.6.30.9/include";make install)
-Ferdy
|
