Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NTop: Misc

PF_RING huge drops

 

 

NTop misc RSS feed   Index | Next | Previous | View Threaded


fedor.sakharov at gmail

Oct 29, 2009, 3:07 AM

Post #1 of 8 (1897 views)
Permalink
PF_RING huge drops

Hello.

I have been testing libpcap and pf_ring recently. And the results have
been significantly different from those mentioned in pf_ring papers. I
have tested CPU load, interface drop and a packet capture drop.

Hardware used:

three SUN x2250 servers with

CPU: Intel(R) Xeon(R) CPU           X5460  @ 3.16GHz
Memory: 8Gb
Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)

Software used:

Kernel: 2.6.26-amd64
PF_RING: 4.0 (compiled as a kernel module)
libpcap: 0.8
libpcap+mmap: 0.9.8

first node runs PF_RING
second node runs libpcap
and the third node runs libpcap+mmap.

Tests carried out:
I have been generating traffic with pktgen tool with different speeds
and packet sizes. Here are the results of tests with 64 byte packets.
The difference in performance between libpcap and libpfring is huge.

libpcap:
55 Mbits/sec -- 50% CPU load, 0% dropped by the interface, 0% dropped
by libpcap.
77 Mbits/sec -- 60% CPU load, 0% dropped by the interface, <1% dropped
by libpcap.
94 Mbits/sec -- 66% CPU load, 0% dropped by the interface, <1% dropped
by libpcap.
140 Mbits/sec -- 80% CPU load, 0% dropped by the interface, <1%
dropped by libpcap.
406 Mbits/sec -- 110% CPU load, 4% dropped by the interface, 2%
dropped by libpcap.
726 Mbits/sec -- 160% CPU load, 28% dropped by the interface, 10%
dropped by libpcap.

libpfring:
55 Mbits/sec --12% CPU load, 0% dropped by the interface, 0% dropped
by libpfring.
77 Mbits/sec -- 22% CPU load, 0% dropped by the interface, 0% dropped
by libpfring.
94 Mbits/sec -- 35% CPU load, 0% dropped by the interface, 0% dropped
by libpfring.
140 Mbits/sec -- 141% CPU load, 22% dropped by the interface, 0%
dropped by libpfring.
406 Mbits/sec -- 153% CPU load, 71% dropped by the interface, <1%
dropped by libpfring.
726 Mbits/sec -- 170% CPU load, 83% dropped by the interface, 1%
dropped by libpfring.

As you see the results differ a lot even on 140 Mbits/sec and when
pf_ring is used interface drops and CPU load are huge. Is there
anything that can be done to solve this problem?

--
Best regards,
Fedor Sakharov
_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc


bcronje at gmail

Oct 29, 2009, 3:30 AM

Post #2 of 8 (1843 views)
Permalink
Re: PF_RING huge drops [In reply to]

Hi Fedor,

Out of interest sake, what are the stats for pf_ring 3.x ? As per a previous
post by Luca non-transparent mode has not been implemented in 4.0 yet, so
the kernel is processing the packets as well which will add to CPU load.

Also what are your interrupt coalescence settings, NIC RX ring size and
pf_ring size?

Beyers

On Thu, Oct 29, 2009 at 12:07 PM, Fedor Sakharov
<fedor.sakharov [at] gmail>wrote:

> Hello.
>
> I have been testing libpcap and pf_ring recently. And the results have
> been significantly different from those mentioned in pf_ring papers. I
> have tested CPU load, interface drop and a packet capture drop.
>
> Hardware used:
>
> three SUN x2250 servers with
>
> CPU: Intel(R) Xeon(R) CPU X5460 @ 3.16GHz
> Memory: 8Gb
> Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
> Controller (rev 06)
>
> Software used:
>
> Kernel: 2.6.26-amd64
> PF_RING: 4.0 (compiled as a kernel module)
> libpcap: 0.8
> libpcap+mmap: 0.9.8
>
> first node runs PF_RING
> second node runs libpcap
> and the third node runs libpcap+mmap.
>
> Tests carried out:
> I have been generating traffic with pktgen tool with different speeds
> and packet sizes. Here are the results of tests with 64 byte packets.
> The difference in performance between libpcap and libpfring is huge.
>
> libpcap:
> 55 Mbits/sec -- 50% CPU load, 0% dropped by the interface, 0% dropped
> by libpcap.
> 77 Mbits/sec -- 60% CPU load, 0% dropped by the interface, <1% dropped
> by libpcap.
> 94 Mbits/sec -- 66% CPU load, 0% dropped by the interface, <1% dropped
> by libpcap.
> 140 Mbits/sec -- 80% CPU load, 0% dropped by the interface, <1%
> dropped by libpcap.
> 406 Mbits/sec -- 110% CPU load, 4% dropped by the interface, 2%
> dropped by libpcap.
> 726 Mbits/sec -- 160% CPU load, 28% dropped by the interface, 10%
> dropped by libpcap.
>
> libpfring:
> 55 Mbits/sec --12% CPU load, 0% dropped by the interface, 0% dropped
> by libpfring.
> 77 Mbits/sec -- 22% CPU load, 0% dropped by the interface, 0% dropped
> by libpfring.
> 94 Mbits/sec -- 35% CPU load, 0% dropped by the interface, 0% dropped
> by libpfring.
> 140 Mbits/sec -- 141% CPU load, 22% dropped by the interface, 0%
> dropped by libpfring.
> 406 Mbits/sec -- 153% CPU load, 71% dropped by the interface, <1%
> dropped by libpfring.
> 726 Mbits/sec -- 170% CPU load, 83% dropped by the interface, 1%
> dropped by libpfring.
>
> As you see the results differ a lot even on 140 Mbits/sec and when
> pf_ring is used interface drops and CPU load are huge. Is there
> anything that can be done to solve this problem?
>
> --
> Best regards,
> Fedor Sakharov
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>


deri at ntop

Oct 29, 2009, 6:13 AM

Post #3 of 8 (1837 views)
Permalink
Re: PF_RING huge drops [In reply to]

Fedor
as Beyers said, the new v4 is easier to use as it does not require a kernel patch but at the same time due to the way it hooks to the kernel it will be slower than 3.x but in the worst case it should be as fast as pcap.

What application have you used for capturing data? You can use PF_RING/userland/examples/pfcount (pure PF_RING application) or
PF_RING/userland/examples/pcount (libpcap application with PF_RING support).

Cheers Luca


On 10/29/2009 11:30 AM, Beyers Cronje wrote:
Hi Fedor,

Out of interest sake, what are the stats for pf_ring 3.x ? As per a previous post by Luca non-transparent mode has not been implemented in 4.0 yet, so the kernel is processing the packets as well which will add to CPU load.

Also what are your interrupt coalescence settings, NIC RX ring size and pf_ring size?

Beyers

On Thu, Oct 29, 2009 at 12:07 PM, Fedor Sakharov <fedor.sakharov [at] gmail> wrote:
Hello.

I have been testing libpcap and pf_ring recently. And the results have
been significantly different from those mentioned in pf_ring papers. I
have tested CPU load, interface drop and a packet capture drop.

Hardware used:

three SUN x2250 servers with

CPU: Intel(R) Xeon(R) CPU X5460 @ 3.16GHz
Memory: 8Gb
Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)

Software used:

Kernel: 2.6.26-amd64
PF_RING: 4.0 (compiled as a kernel module)
libpcap: 0.8
libpcap+mmap: 0.9.8

first node runs PF_RING
second node runs libpcap
and the third node runs libpcap+mmap.

Tests carried out:
I have been generating traffic with pktgen tool with different speeds
and packet sizes. Here are the results of tests with 64 byte packets.
The difference in performance between libpcap and libpfring is huge.

libpcap:
55 Mbits/sec -- 50% CPU load, 0% dropped by the interface, 0% dropped
by libpcap.
77 Mbits/sec -- 60% CPU load, 0% dropped by the interface, <1% dropped
by libpcap.
94 Mbits/sec -- 66% CPU load, 0% dropped by the interface, <1% dropped
by libpcap.
140 Mbits/sec -- 80% CPU load, 0% dropped by the interface, <1%
dropped by libpcap.
406 Mbits/sec -- 110% CPU load, 4% dropped by the interface, 2%
dropped by libpcap.
726 Mbits/sec -- 160% CPU load, 28% dropped by the interface, 10%
dropped by libpcap.

libpfring:
55 Mbits/sec --12% CPU load, 0% dropped by the interface, 0% dropped
by libpfring.
77 Mbits/sec -- 22% CPU load, 0% dropped by the interface, 0% dropped
by libpfring.
94 Mbits/sec -- 35% CPU load, 0% dropped by the interface, 0% dropped
by libpfring.
140 Mbits/sec -- 141% CPU load, 22% dropped by the interface, 0%
dropped by libpfring.
406 Mbits/sec -- 153% CPU load, 71% dropped by the interface, <1%
dropped by libpfring.
726 Mbits/sec -- 170% CPU load, 83% dropped by the interface, 1%
dropped by libpfring.

As you see the results differ a lot even on 140 Mbits/sec and when
pf_ring is used interface drops and CPU load are huge. Is there
anything that can be done to solve this problem?

--
Best regards,
Fedor Sakharov
_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc"]http://listgateway.unipi.it/mailman/listinfo/ntop-misc


_______________________________________________ Ntop-misc mailing list Ntop-misc [at] listgateway http://listgateway.unipi.it/mailman/listinfo/ntop-misc"]http://listgateway.unipi.it/mailman/listinfo/ntop-misc


fedor.sakharov at gmail

Nov 3, 2009, 6:33 AM

Post #4 of 8 (1795 views)
Permalink
Re: PF_RING huge drops [In reply to]

>Hi Fedor,
>
>Out of interest sake, what are the stats for pf_ring 3.x ? As per a previous
>post by Luca non-transparent mode has not been implemented in 4.0 yet, so
>the kernel is processing the packets as well which will add to CPU load.
>
>Also what are your interrupt coalescence settings, NIC RX ring size and
>pf_ring size?

Here are the coalescence settings:

Coalesce parameters for eth0:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 3
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 0

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

and the info on the PF_RING:

PF_RING Version : 4.0.0 ($Revision: 3966 $)
Ring slots : 8112
Slot version : 10
Capture TX : Yes [RX+TX]
IP Defragment : No
Transparent mode : Yes
Total rings : 0
Total plugins : 0

>Beyers
>

The stats for 3.x are practically almost the same.
--
Best regards,
Fedor Sakharov
_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc


bcronje at gmail

Nov 3, 2009, 6:42 AM

Post #5 of 8 (1795 views)
Permalink
Re: PF_RING huge drops [In reply to]

Hi,

And the output of 'ethtool -g' ?

On Tue, Nov 3, 2009 at 4:33 PM, Fedor Sakharov <fedor.sakharov [at] gmail>wrote:

> >Hi Fedor,
> >
> >Out of interest sake, what are the stats for pf_ring 3.x ? As per a
> previous
> >post by Luca non-transparent mode has not been implemented in 4.0 yet, so
> >the kernel is processing the packets as well which will add to CPU load.
> >
> >Also what are your interrupt coalescence settings, NIC RX ring size and
> >pf_ring size?
>
> Here are the coalescence settings:
>
> Coalesce parameters for eth0:
> Adaptive RX: off TX: off
> stats-block-usecs: 0
> sample-interval: 0
> pkt-rate-low: 0
> pkt-rate-high: 0
>
> rx-usecs: 3
> rx-frames: 0
> rx-usecs-irq: 0
> rx-frames-irq: 0
>
> tx-usecs: 0
> tx-frames: 0
> tx-usecs-irq: 0
> tx-frames-irq: 0
>
> rx-usecs-low: 0
> rx-frame-low: 0
> tx-usecs-low: 0
> tx-frame-low: 0
>
> rx-usecs-high: 0
> rx-frame-high: 0
> tx-usecs-high: 0
> tx-frame-high: 0
>
> and the info on the PF_RING:
>
> PF_RING Version : 4.0.0 ($Revision: 3966 $)
> Ring slots : 8112
> Slot version : 10
> Capture TX : Yes [RX+TX]
> IP Defragment : No
> Transparent mode : Yes
> Total rings : 0
> Total plugins : 0
>
> >Beyers
> >
>
> The stats for 3.x are practically almost the same.
> --
> Best regards,
> Fedor Sakharov
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>


gamajun at cs

Nov 3, 2009, 6:51 AM

Post #6 of 8 (1793 views)
Permalink
Re: PF_RING huge drops [In reply to]

Hi, I'll answer for Fedor. Here is ethtool output for this particular
interface:
~ # ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 256

Regards,

Dennis

Beyers Cronje пишет:
> Hi,
>
> And the output of 'ethtool -g' ?
>
> On Tue, Nov 3, 2009 at 4:33 PM, Fedor Sakharov
> <fedor.sakharov [at] gmail <mailto:fedor.sakharov [at] gmail>> wrote:
>
> >Hi Fedor,
> >
> >Out of interest sake, what are the stats for pf_ring 3.x ? As per
> a previous
> >post by Luca non-transparent mode has not been implemented in 4.0
> yet, so
> >the kernel is processing the packets as well which will add to
> CPU load.
> >
> >Also what are your interrupt coalescence settings, NIC RX ring
> size and
> >pf_ring size?
>
> Here are the coalescence settings:
>
> Coalesce parameters for eth0:
> Adaptive RX: off TX: off
> stats-block-usecs: 0
> sample-interval: 0
> pkt-rate-low: 0
> pkt-rate-high: 0
>
> rx-usecs: 3
> rx-frames: 0
> rx-usecs-irq: 0
> rx-frames-irq: 0
>
> tx-usecs: 0
> tx-frames: 0
> tx-usecs-irq: 0
> tx-frames-irq: 0
>
> rx-usecs-low: 0
> rx-frame-low: 0
> tx-usecs-low: 0
> tx-frame-low: 0
>
> rx-usecs-high: 0
> rx-frame-high: 0
> tx-usecs-high: 0
> tx-frame-high: 0
>
> and the info on the PF_RING:
>
> PF_RING Version : 4.0.0 ($Revision: 3966 $)
> Ring slots : 8112
> Slot version : 10
> Capture TX : Yes [RX+TX]
> IP Defragment : No
> Transparent mode : Yes
> Total rings : 0
> Total plugins : 0
>
> >Beyers
> >
>
> The stats for 3.x are practically almost the same.
> --
> Best regards,
> Fedor Sakharov
>

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc


bcronje at gmail

Nov 3, 2009, 7:02 AM

Post #7 of 8 (1790 views)
Permalink
Re: PF_RING huge drops [In reply to]

Since you seeing a lot of device drops I'd definately increase the device RX
ring size to 4096. That should already make a difference.

Also definately set pf_ring transparent mode off. ' transparent_mode=0
enable_tx_capture=0'

If the increase is not enough I'd play around with different interrupt
coalescing settings to achieve optimal performance.

Beyers

On Tue, Nov 3, 2009 at 4:51 PM, Dennis Gamayunov <gamajun [at] cs> wrote:

> Hi, I'll answer for Fedor. Here is ethtool output for this particular
> interface:
> ~ # ethtool -g eth0
> Ring parameters for eth0:
> Pre-set maximums:
> RX: 4096
> RX Mini: 0
> RX Jumbo: 0
> TX: 4096
> Current hardware settings:
> RX: 256
> RX Mini: 0
> RX Jumbo: 0
> TX: 256
>
> Regards,
>
> Dennis
>
> Beyers Cronje пишет:
> > Hi,
> >
> > And the output of 'ethtool -g' ?
> >
> > On Tue, Nov 3, 2009 at 4:33 PM, Fedor Sakharov
> > <fedor.sakharov [at] gmail <mailto:fedor.sakharov [at] gmail>> wrote:
> >
> > >Hi Fedor,
> > >
> > >Out of interest sake, what are the stats for pf_ring 3.x ? As per
> > a previous
> > >post by Luca non-transparent mode has not been implemented in 4.0
> > yet, so
> > >the kernel is processing the packets as well which will add to
> > CPU load.
> > >
> > >Also what are your interrupt coalescence settings, NIC RX ring
> > size and
> > >pf_ring size?
> >
> > Here are the coalescence settings:
> >
> > Coalesce parameters for eth0:
> > Adaptive RX: off TX: off
> > stats-block-usecs: 0
> > sample-interval: 0
> > pkt-rate-low: 0
> > pkt-rate-high: 0
> >
> > rx-usecs: 3
> > rx-frames: 0
> > rx-usecs-irq: 0
> > rx-frames-irq: 0
> >
> > tx-usecs: 0
> > tx-frames: 0
> > tx-usecs-irq: 0
> > tx-frames-irq: 0
> >
> > rx-usecs-low: 0
> > rx-frame-low: 0
> > tx-usecs-low: 0
> > tx-frame-low: 0
> >
> > rx-usecs-high: 0
> > rx-frame-high: 0
> > tx-usecs-high: 0
> > tx-frame-high: 0
> >
> > and the info on the PF_RING:
> >
> > PF_RING Version : 4.0.0 ($Revision: 3966 $)
> > Ring slots : 8112
> > Slot version : 10
> > Capture TX : Yes [RX+TX]
> > IP Defragment : No
> > Transparent mode : Yes
> > Total rings : 0
> > Total plugins : 0
> >
> > >Beyers
> > >
> >
> > The stats for 3.x are practically almost the same.
> > --
> > Best regards,
> > Fedor Sakharov
> >
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>


deri at ntop

Nov 5, 2009, 12:19 AM

Post #8 of 8 (1766 views)
Permalink
Re: PF_RING huge drops [In reply to]

Fedor
please first move to PF_RING 4.1 and then re-run your tests

Luca

On 11/03/2009 03:33 PM, Fedor Sakharov wrote:
Hi Fedor, Out of interest sake, what are the stats for pf_ring 3.x ? As per a previous post by Luca non-transparent mode has not been implemented in 4.0 yet, so the kernel is processing the packets as well which will add to CPU load. Also what are your interrupt coalescence settings, NIC RX ring size and pf_ring size?
Here are the coalescence settings: Coalesce parameters for eth0: Adaptive RX: off TX: off stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0 rx-usecs: 3 rx-frames: 0 rx-usecs-irq: 0 rx-frames-irq: 0 tx-usecs: 0 tx-frames: 0 tx-usecs-irq: 0 tx-frames-irq: 0 rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0 rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0 and the info on the PF_RING: PF_RING Version : 4.0.0 ($Revision: 3966 $) Ring slots : 8112 Slot version : 10 Capture TX : Yes [RX+TX] IP Defragment : No Transparent mode : Yes Total rings : 0 Total plugins : 0
Beyers
The stats for 3.x are practically almost the same.

NTop misc RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.