Mailing List Archive: NANOG: users

Software router state of the art

1 2 3

View All

Index | Next | Previous | View Threaded

zzuser at yahoo

Jul 23, 2008, 2:52 AM

Post #1 of 75 (1915 views)
Permalink

Software router state of the art

Hi all!

There's been some discussion on the list regarding software routers lately and this piqued my interest. Does anybody have any recent performance and capability statistics (eg. forwarding rates with full BGP tables and N ethernet interfaces) or any pointer to what the current state of art in software routers is?

- Zed

Valdis.Kletnieks at vt

Jul 23, 2008, 8:24 AM

Post #2 of 75 (1879 views)
Permalink

Re: Software router state of the art [In reply to]

On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:
> There's been some discussion on the list regarding software routers

The performance of "software routers" has always had a hardware component.

Basically, for the vast majority of them, take your PCI bus bandwidth,
count how many times a packet has to cross it, and do the math. You can't
forward more than that much traffic no matter *what* software you run on
that box. If that number falls short, stop right there and look for
some box of different design that has the required backplane bandwidth.

You will, of course, take additional performance hits due to locking issues
and similar in your software stack (that, and most "software" routers will
suffer from not having special hardware assist for routing table lookups).

Let us know if you find a suitable chassis/motherboard that has enough
bandwidth to make it worth thinking about for anything other than the
smaller edge routers that most providers have zillions of... :)

charles at thewybles

Jul 23, 2008, 8:36 AM

Post #3 of 75 (1879 views)
Permalink

Re: Software router state of the art [In reply to]

Valdis.Kletnieks [at] vt wrote:
> On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:
>
>> There's been some discussion on the list regarding software routers
>>
>
> The performance of "software routers" has always had a hardware component.
>
> Basically, for the vast majority of them, take your PCI bus bandwidth,
> count how many times a packet has to cross it, and do the math. You can't
> forward more than that much traffic no matter *what* software you run on
> that box. If that number falls short, stop right there and look for
> some box of different design that has the required backplane bandwidth.
>

This might be of interest:

http://nrg.cs.ucl.ac.uk/mjh/tmp/vrouter-perf.pdf

--

Charles Wyble (818) 280 - 7059
http://charlesnw.blogspot.com
CTO Known Element Enterprises / SoCal WiFI project

herrin-nanog at dirtside

Jul 23, 2008, 9:07 AM

Post #4 of 75 (1875 views)
Permalink

Re: Software router state of the art [In reply to]

On Wed, Jul 23, 2008 at 11:24 AM, <Valdis.Kletnieks [at] vt> wrote:
> On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:
>> There's been some discussion on the list regarding software routers
>
> The performance of "software routers" has always had a hardware component.
>
> Basically, for the vast majority of them, take your PCI bus bandwidth,
> count how many times a packet has to cross it, and do the math. You can't
> forward more than that much traffic no matter *what* software you run on
> that box. If that number falls short, stop right there and look for
> some box of different design that has the required backplane bandwidth.

The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.

Regards,
Bill Herrin

--
William D. Herrin ................ herrin [at] dirtside bill [at] herrin
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

adrian at creative

Jul 23, 2008, 9:17 AM

Post #5 of 75 (1876 views)
Permalink

Re: Software router state of the art [In reply to]

On Wed, Jul 23, 2008, Charles Wyble wrote:

> This might be of interest:
>
> http://nrg.cs.ucl.ac.uk/mjh/tmp/vrouter-perf.pdf

Various FreeBSD related guys are working on parallelising the forwarding
layer enough to use the multiple tx/rx queues in some chipsets such as the
Intel gig/10ge stuff.

1 mil pps has been broken that way, but it uses lots of cores to get there.
(8, I think?)

Linux apparently is/has headed down this path.

If someone were to spend some time dissecting the rest of the code to
also optimise the single-core throughput then you may see some interesting
software routers using commodity hardware (for values of "commodity"
roughly equal to "PC servers", rather than "magic lotsacore core MIPS with
some extra glue for jacking packets around."

Sure its not a CRS-1, but reliably doing a mil pps with a smattering of
low-touch features would be rather useful, no?

(Then, add say, l2tp/ppp into that mix, just as a crazy on-topic example..)

Adrian

cmarlatt at rxsec

Jul 23, 2008, 9:25 AM

Post #6 of 75 (1893 views)
Permalink

Re: Software router state of the art [In reply to]

Adrian Chadd wrote:
>
> 1 mil pps has been broken that way, but it uses lots of cores to get there.
> (8, I think?)
>

http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-06/msg00364.html
has all the details. It's rather long thread but 1mpps was achieved on a
single cpu IIRC (the server had multiple cpus but only one being used
for forwarding). Firewall rules slowed it down quite a bit but theres
also some work out there being done to minimize this.

Regards,

Chris

adrian at creative

Jul 23, 2008, 9:32 AM

Post #7 of 75 (1879 views)
Permalink

Re: Software router state of the art [In reply to]

On Wed, Jul 23, 2008, Chris Marlatt wrote:

> http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-06/msg00364.html
> has all the details. It's rather long thread but 1mpps was achieved on a
> single cpu IIRC (the server had multiple cpus but only one being used
> for forwarding). Firewall rules slowed it down quite a bit but theres
> also some work out there being done to minimize this.

Yah, all of that is happening. Some people keep asking why FreeBSD-4
forwarding was always much faster than same-hardware forwarding under
current FreeBSD but at least thats finally being worked on.

Of course, with my FreeBSD advocacy hat on, if you -want- to see
something like FreeBSD handle 1mil+ pps forwarding then you should
really drop the FreeBSD Foundation a line and introduce yourself.
There are developers working on this (note: not me! :) who would
benefit from equipment and funding.

Anyway. Some PC class hardware is pretty damned fast. Some vendors
even build highish-throughput firewalls and proxies out of PC class
hardware. :) The "wah wah PC class hardware has anemic bus IO/memory IO/
CPU speed/ethernet modules and is thus too crap for serious routing" argument
is pretty much over for at least 1 mil pps, perhaps more.

2c,

Adrian

joelja at bogus

Jul 23, 2008, 9:50 AM

Post #8 of 75 (1875 views)
Permalink

Re: Software router state of the art [In reply to]

Valdis.Kletnieks [at] vt wrote:
> On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:
>> There's been some discussion on the list regarding software routers
>
> The performance of "software routers" has always had a hardware component.
>
> Basically, for the vast majority of them, take your PCI bus bandwidth,
> count how many times a packet has to cross it, and do the math. You can't
> forward more than that much traffic no matter *what* software you run on
> that box. If that number falls short, stop right there and look for
> some box of different design that has the required backplane bandwidth.
>
> You will, of course, take additional performance hits due to locking issues
> and similar in your software stack (that, and most "software" routers will
> suffer from not having special hardware assist for routing table lookups).

The current state of the art is around 2 million pps for fast intel arch
system.

> Let us know if you find a suitable chassis/motherboard that has enough
> bandwidth to make it worth thinking about for anything other than the
> smaller edge routers that most providers have zillions of... :)
>

nanog at data102

Jul 23, 2008, 10:46 AM

Post #9 of 75 (1861 views)
Permalink

Re: Software router state of the art [In reply to]

That is a very interesting paper. Seriously, 7mpps with an
off-the-shelf Dell 2950? Even if it were -half- that throughput, for a
pure ethernet forwarding solution that is incredible. Shoot, buy a
handful of them as hot spares and still save a bundle.

Highly recommended reading, even if (like me) you're anti-commodity routing.

Cheers,
Randal

On Wed, Jul 23, 2008 at 10:17 AM, Adrian Chadd <adrian [at] creative> wrote:
> On Wed, Jul 23, 2008, Charles Wyble wrote:
>
>> This might be of interest:
>>
>> http://nrg.cs.ucl.ac.uk/mjh/tmp/vrouter-perf.pdf
>
> Various FreeBSD related guys are working on parallelising the forwarding
> layer enough to use the multiple tx/rx queues in some chipsets such as the
> Intel gig/10ge stuff.
>
> 1 mil pps has been broken that way, but it uses lots of cores to get there.
> (8, I think?)
>
> Linux apparently is/has headed down this path.

lists at memetic

Jul 23, 2008, 11:00 AM

Post #10 of 75 (1870 views)
Permalink

Re: Software router state of the art [In reply to]

Adrian Chadd wrote:
> On Wed, Jul 23, 2008, Charles Wyble wrote:
>
>
> Sure its not a CRS-1, but reliably doing a mil pps with a smattering of
> low-touch features would be rather useful, no?
>
> (Then, add say, l2tp/ppp into that mix, just as a crazy on-topic example..)
>
Sounds like a Juniper J-series. Have a look at the forwarding figures
for the J6350. It does something around 2mpps and it's just an intel CPU
with some PCI/PCI-X interfaces. The device just below it, the J4350 uses
a 2.53Ghz celeron. I'm not sure what the J6350 uses.

adam.

naveen at calpop

Jul 23, 2008, 11:05 AM

Post #11 of 75 (1853 views)
Permalink

Re: Software router state of the art [In reply to]

> The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
> the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.

I wonder, has anyone heard of this used for IDS? I've been looking at
building a commodity SNORT solution, and wondering if a powerful network
card will help, or would the bottleneck be in processing the packets and
overhead from the OS?

- naveen

cmadams at hiwaay

Jul 23, 2008, 11:16 AM

Post #12 of 75 (1861 views)
Permalink

Re: Software router state of the art [In reply to]

Once upon a time, Adam Armstrong <lists [at] memetic> said:
> Sounds like a Juniper J-series. Have a look at the forwarding figures
> for the J6350. It does something around 2mpps and it's just an intel CPU
> with some PCI/PCI-X interfaces. The device just below it, the J4350 uses
> a 2.53Ghz celeron. I'm not sure what the J6350 uses.

IIRC the new slots (the EPIMs) are PCI-E. The J6350 CPU is a P4 3.4GHz.
It is my understanding that the J-series use a real-time layer under the
FreeBSD kernel and have a real-time thread for forwarding (as opposed to
the M-series with a hardware forwarding engine).

--
Chris Adams <cmadams [at] hiwaay>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

herrin-nanog at dirtside

Jul 23, 2008, 11:17 AM

Post #13 of 75 (1867 views)
Permalink

Re: Software router state of the art [In reply to]

On Wed, Jul 23, 2008 at 2:03 PM, Naveen Nathan <naveen [at] lastninja> wrote:
>> The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
>> the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.
>
> I wonder, has anyone heard of this used for IDS? I've been looking at
> building a commodity SNORT solution, and wondering if a powerful network
> card will help, or would the bottleneck be in processing the packets and
> overhead from the OS?

The first bottleneck is the interrupts from the NIC. With a generic
Intel NIC under Linux, you start to lose a non-trivial number of
packets around 700mbps of "normal" traffic because it can't service
the interrupts quickly enough.

The DAG card can be dropped in to replace the interface used for a
libpcap-based application. When I tested the 1gbps PCIE version, I
lost no packets to 1gbps and my capture application's CPU usage
dropped to about 1/5th of what it was with the generic NIC. YMMV.

Regards,
Bill Herrin

--
William D. Herrin ................ herrin [at] dirtside bill [at] herrin
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

morrowc.lists at gmail

Jul 23, 2008, 11:21 AM

Post #14 of 75 (1857 views)
Permalink

Re: Software router state of the art [In reply to]

On Wed, Jul 23, 2008 at 11:05 AM, Naveen Nathan <naveen [at] calpop> wrote:
>> The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
>> the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.
>
> I wonder, has anyone heard of this used for IDS? I've been looking at
> building a commodity SNORT solution, and wondering if a powerful network
> card will help, or would the bottleneck be in processing the packets and
> overhead from the OS?

http://www.endace.com/our-products/ninja-appliances/NinjaProbe-NIDS

snort at 1g & 10g

-chris

wcyoung at buffalo

Jul 23, 2008, 12:05 PM

Post #15 of 75 (1871 views)
Permalink

Re: Software router state of the art [In reply to]

We use them here and there (the 1Gig versions). The biggest thing to
think about is the types of rule-sets you'll be using compounded by
the number of flows being created / expired. Once tuned, they work
quite well, but the balance is how fast you can pull/analyze out of
RAM. Compiling the rules down to the card's level speeds things up a
bit, but at the loss of using more dynamic rulesets.

If you can get the raw data to some sort of larger medium (say,
rotating pcaps on a disk), you length the buffer-window. FWIW however,
probably the best way to scale this is get an Xport fiber regen tap,
populate with a few of these, tune them to monitor different segments
based on address space or port ranges. You'll have yourself a
relatively cheap solution, but extremely effective solution.

I've yet to test out the NinjaProbes... It's on my todo list...

On Jul 23, 2008, at 2:21 PM, Christopher Morrow wrote:

> On Wed, Jul 23, 2008 at 11:05 AM, Naveen Nathan <naveen [at] calpop>
> wrote:
>>> The Endace DAG cards claim they can move 7 gbps over a PCI-X bus
>>> from
>>> the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.
>>
>> I wonder, has anyone heard of this used for IDS? I've been looking at
>> building a commodity SNORT solution, and wondering if a powerful
>> network
>> card will help, or would the bottleneck be in processing the
>> packets and
>> overhead from the OS?
>
> http://www.endace.com/our-products/ninja-appliances/NinjaProbe-NIDS
>
> snort at 1g & 10g
>
> -chris
>

--
Wes Young
Network Security Analyst
CIT - University at Buffalo
http://claimid.com/saxjazman9

Attachments:

smime.p7s (2.39 KB)

jeff at ocjtech

Jul 23, 2008, 12:50 PM

Post #16 of 75 (1847 views)
Permalink

Re: Software router state of the art [In reply to]

On Wed, Jul 23, 2008 at 11:17 AM, Adrian Chadd <adrian [at] creative> wrote:
>
> Various FreeBSD related guys are working on parallelising the forwarding
> layer enough to use the multiple tx/rx queues in some chipsets such as the
> Intel gig/10ge stuff.
>
> Linux apparently is/has headed down this path.

I've seen some notes from Dave Miller about adding multiple TX queues
to the 2.6.27 kernel. See Dave's blog for the gory details:

http://vger.kernel.org/~davem/cgi-bin/blog.cgi
http://git.kernel.org/?p=linux/kernel/git/davem/net-tx-2.6.git;a=summary

AFAIK he hasn't made any claims about performance improvements. I
don't know the state of RX queues in Linux.

Jeff

oberman at es

Jul 23, 2008, 12:59 PM

Post #17 of 75 (1847 views)
Permalink

Re: Software router state of the art [In reply to]

> Date: Wed, 23 Jul 2008 14:17:53 -0400
> From: "William Herrin" <herrin-nanog [at] dirtside>
>
> On Wed, Jul 23, 2008 at 2:03 PM, Naveen Nathan <naveen [at] lastninja> wrote:
> >> The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
> >> the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.
> >
> > I wonder, has anyone heard of this used for IDS? I've been looking at
> > building a commodity SNORT solution, and wondering if a powerful network
> > card will help, or would the bottleneck be in processing the packets and
> > overhead from the OS?
>
> The first bottleneck is the interrupts from the NIC. With a generic
> Intel NIC under Linux, you start to lose a non-trivial number of
> packets around 700mbps of "normal" traffic because it can't service
> the interrupts quickly enough.

Most modern high performance network cards support MSI (Message Signaled
Interrupts) which generate real interrupts only in an intelligent
basis. and only at a controlled rate. Windows, Solaris and FreeBSD have
support for MSI and I think Linux does, too. It requires both hardware
and software support.

With MSI, TSO, LRO, and PCI-E with hardware that supports these, 9.5
Gbps TCP flows between systems is possible with minimal tuning. That
puts the bottleneck back on the forwarding software in the CPU to do
the forwarding at high rates.
--
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: oberman [at] es Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751

herrin-nanog at dirtside

Jul 23, 2008, 1:51 PM

Post #18 of 75 (1846 views)
Permalink

Re: Software router state of the art [In reply to]

On Wed, Jul 23, 2008 at 3:59 PM, Kevin Oberman <oberman [at] es> wrote:
>> The first bottleneck is the interrupts from the NIC. With a generic
>> Intel NIC under Linux, you start to lose a non-trivial number of
>> packets around 700mbps of "normal" traffic because it can't service
>> the interrupts quickly enough.
>
> Most modern high performance network cards support MSI (Message Signaled
> Interrupts) which generate real interrupts only in an intelligent
> basis. and only at a controlled rate. Windows, Solaris and FreeBSD have
> support for MSI and I think Linux does, too. It requires both hardware
> and software support.

"ethtool -c". Thanks Sargun for putting me on to "I/O Coalescing."

But cards like the Intel Pro/1000 have 64k of memory for buffering
packets, both in and out. Few have very much more than 64k. 64k means
32k to tx and 32k to rx. Means you darn well better generate an
interrupt when you get near 16k so that you don't fill the buffer
before the 16k you generated the interrupt for has been cleared. Means
you're generating an interrupt at least for every 10 or so 1500 byte
packets.

Regards,
Bill

--
William D. Herrin ................ herrin [at] dirtside bill [at] herrin
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

oberman at es

Jul 23, 2008, 2:23 PM

Post #19 of 75 (1845 views)
Permalink

Re: Software router state of the art [In reply to]

> Date: Wed, 23 Jul 2008 16:51:50 -0400
> From: "William Herrin" <herrin-nanog [at] dirtside>
> Sender: wherrin [at] gmail
>
> On Wed, Jul 23, 2008 at 3:59 PM, Kevin Oberman <oberman [at] es> wrote:
> >> The first bottleneck is the interrupts from the NIC. With a generic
> >> Intel NIC under Linux, you start to lose a non-trivial number of
> >> packets around 700mbps of "normal" traffic because it can't service
> >> the interrupts quickly enough.
> >
> > Most modern high performance network cards support MSI (Message Signaled
> > Interrupts) which generate real interrupts only in an intelligent
> > basis. and only at a controlled rate. Windows, Solaris and FreeBSD have
> > support for MSI and I think Linux does, too. It requires both hardware
> > and software support.
>
> "ethtool -c". Thanks Sargun for putting me on to "I/O Coalescing."
>
> But cards like the Intel Pro/1000 have 64k of memory for buffering
> packets, both in and out. Few have very much more than 64k. 64k means
> 32k to tx and 32k to rx. Means you darn well better generate an
> interrupt when you get near 16k so that you don't fill the buffer
> before the 16k you generated the interrupt for has been cleared. Means
> you're generating an interrupt at least for every 10 or so 1500 byte
> packets.

You have just hit on a huge problems with most (all?) 1G and 10G
hardware. The buffers are way too small for optimal performance in any
case where the RTT is anything more that half a millisecond, you exhaust
the window and stall the stream.

I need port move multi-gigabit streams across the country and between the
US and Europe. Those are a bit too far apart for those tiny buffers to
be of any use at all. This would require 3 GB of buffers. This same
problem also make TCP off-load of no use at all.
--
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: oberman [at] es Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751

tims at donet

Jul 24, 2008, 5:25 AM

Post #20 of 75 (1855 views)
Permalink

RE: Software router state of the art [In reply to]

Is anyone using Vyatta for routing? I sure would like to know about any experience with it in production.

http://www.vyatta.com/

--
Tim Sanderson, network administrator
tims [at] donet

-----Original Message-----
From: randal k [mailto:nanog [at] data102]
Sent: Wednesday, July 23, 2008 1:46 PM
To: Adrian Chadd
Cc: nanog [at] merit
Subject: Re: Software router state of the art

That is a very interesting paper. Seriously, 7mpps with an
off-the-shelf Dell 2950? Even if it were -half- that throughput, for a
pure ethernet forwarding solution that is incredible. Shoot, buy a
handful of them as hot spares and still save a bundle.

Highly recommended reading, even if (like me) you're anti-commodity routing.

Cheers,
Randal

On Wed, Jul 23, 2008 at 10:17 AM, Adrian Chadd <adrian [at] creative> wrote:
> On Wed, Jul 23, 2008, Charles Wyble wrote:
>
>> This might be of interest:
>>
>> http://nrg.cs.ucl.ac.uk/mjh/tmp/vrouter-perf.pdf
>
> Various FreeBSD related guys are working on parallelising the forwarding
> layer enough to use the multiple tx/rx queues in some chipsets such as the
> Intel gig/10ge stuff.
>
> 1 mil pps has been broken that way, but it uses lots of cores to get there.
> (8, I think?)
>
> Linux apparently is/has headed down this path.

sharp at sharpone

Jul 25, 2008, 8:44 AM

Post #21 of 75 (1835 views)
Permalink

Re: Software router state of the art [In reply to]

Yes. We put in some Vyatta routers to extend our corporate network into
another building as a temporary solution (the building had a very short
lease, so our boss didn't want to spend any money on Juniper which is
our usual net gear vendor). Consequently, we are still there.. go figure.

When we started w/ them, they were still using the XORP routing engine
(and we haven't upgraded to the new platform yet). My experience wasn't
terribly good. The first issue was a bad memory leak in the router
manager process when VRRP hello times were set to 1 second. The first
indication of something wrong is that our master router crashed,
followed by his backup. Had to physically reboot the boxes to get them
back online, which involved driving there as no one onsite had access to
the cage at the office. All voice and data ran through these routers,
basically rendering every employee useless until we got it back online.
It wasn't a happy day. After that we had to monitor memory and do
controlled reboots every month or so. We eventually convinced Vyatta of
this memory leak and they were able to fix it, but that was a very
frustrating process, and time consuming for us, which is why the next
problems I describe, we have just found our own workarounds.

The next problem was a combination of a problem with the Vyatta and a
problem w/ our IP phones. The Vyatta was sending garp's for the data
vrrp address out the voice vlan (same 2 routers are default gate on both
data and voice vlans). All of the workstations run through the phones
(which sit tagged on voice vlan, and pass traffic from workstation
untagged to data vlan). The phone, seeing the arp for the data vrrp
address on its voice vlan, would send traffic to that address out the
voice vlan, effectively taking that workstation off the net for anything
other than local traffic. That was a bugger to figure out, and basically
we solved it with an arptables rule on the vyatta's. That was the one
advantage of using a Linux (debian) based router platform, was that we
could load other 'unsupported' packages to solve problems like this.

The last thing is that OSPF never really converges correctly. You can
view the OSPF database, and see which default the routers should
converge to, but they do not. They will sit converged to one path for a
while, and if you reboot the other router that generates default, they
will reconverge to it for a while. This hasn't been a big enough problem
for me to worry about it.

Last thing to say is, I haven't tried upgrading since Vyatta abandoned
the XORP platform and moved to the Quagga platform, but I'm guessing
(based on experience w/ Quagga) that they have a lot fewer of these
quirks that I've described.

IMHO, YMMV, etc

--Justin

Tim Sanderson wrote:
> Is anyone using Vyatta for routing? I sure would like to know about any experience with it in production.
>
> http://www.vyatta.com/
>
> --
> Tim Sanderson, network administrator
> tims [at] donet
>
>
> -----Original Message-----
> From: randal k [mailto:nanog [at] data102]
> Sent: Wednesday, July 23, 2008 1:46 PM
> To: Adrian Chadd
> Cc: nanog [at] merit
> Subject: Re: Software router state of the art
>
> That is a very interesting paper. Seriously, 7mpps with an
> off-the-shelf Dell 2950? Even if it were -half- that throughput, for a
> pure ethernet forwarding solution that is incredible. Shoot, buy a
> handful of them as hot spares and still save a bundle.
>
> Highly recommended reading, even if (like me) you're anti-commodity routing.
>
> Cheers,
> Randal
>
> On Wed, Jul 23, 2008 at 10:17 AM, Adrian Chadd <adrian [at] creative> wrote:
>
>> On Wed, Jul 23, 2008, Charles Wyble wrote:
>>
>>
>>> This might be of interest:
>>>
>>> http://nrg.cs.ucl.ac.uk/mjh/tmp/vrouter-perf.pdf
>>>
>> Various FreeBSD related guys are working on parallelising the forwarding
>> layer enough to use the multiple tx/rx queues in some chipsets such as the
>> Intel gig/10ge stuff.
>>
>> 1 mil pps has been broken that way, but it uses lots of cores to get there.
>> (8, I think?)
>>
>> Linux apparently is/has headed down this path.
>>
>
>
>

jgreco at ns

Jul 25, 2008, 9:43 AM

Post #22 of 75 (1802 views)
Permalink

Re: Software router state of the art [In reply to]

> Last thing to say is, I haven't tried upgrading since Vyatta abandoned
> the XORP platform and moved to the Quagga platform, but I'm guessing
> (based on experience w/ Quagga) that they have a lot fewer of these
> quirks that I've described.

Quagga is pretty decent, but it is not uncommon for serious bugs to go
unaddressed for a long time. For example, this bug renders Quagga
nearly unusable for OSPF on FreeBSD 7,

http://bugzilla.quagga.net/show_bug.cgi?id=420

which resulted in some finger-pointing, but the last I heard, it was
due to a kernel interface change where FreeBSD multicast code had been
rewritten and was DTRT, while Linux was doing something else.

This is probably still better than the XORP platform, but it is
unfortunate.

... JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

sdhillon at decarta

Jul 25, 2008, 10:49 AM

Post #23 of 75 (1799 views)
Permalink

Re: Software router state of the art [In reply to]

It would be very useful if there was an effort from the telecom
community to develop a dynamic routing frontend like Quagga. The amount
of human work that it requires in order to build up a product is
enormous. If only someone with millions of dollars could donate
engineers. It would allow the deployment of small branch office systems
at a much lower cost.

Would you rather deploy a $3000 cisco edge box which is a unexpandable,
100 mbit piece of crap, or throw two $2000 Dell boxes and have a 1 GigE
platform?

Joe Greco wrote:
>> Last thing to say is, I haven't tried upgrading since Vyatta abandoned
>> the XORP platform and moved to the Quagga platform, but I'm guessing
>> (based on experience w/ Quagga) that they have a lot fewer of these
>> quirks that I've described.
>>
>
> Quagga is pretty decent, but it is not uncommon for serious bugs to go
> unaddressed for a long time. For example, this bug renders Quagga
> nearly unusable for OSPF on FreeBSD 7,
>
> http://bugzilla.quagga.net/show_bug.cgi?id=420
>
> which resulted in some finger-pointing, but the last I heard, it was
> due to a kernel interface change where FreeBSD multicast code had been
> rewritten and was DTRT, while Linux was doing something else.
>
> This is probably still better than the XORP platform, but it is
> unfortunate.
>
> ... JG
>

--
+1.925.202.9485
Sargun Dhillon
deCarta
sdhillon [at] decarta
www.decarta.com

jgreco at ns

Jul 25, 2008, 10:56 AM

Post #24 of 75 (1802 views)
Permalink

Re: Software router state of the art [In reply to]

> Would you rather deploy a $3000 cisco edge box which is a unexpandable,
> 100 mbit piece of crap, or throw two $2000 Dell boxes and have a 1 GigE
> platform?

You don't need two $2000 Dell boxes to get a 1G platform, but this isn't
the list for that. You also don't need a ton of money to do open source
development (we do one major project here in-house, one that's responsible
for generating a noticeable amount of traffic on most networks). But this
also isn't the list for that discussion.

I suspect software routing is going to continue to be a growing factor in
the packet herding world, as more people want to do more interesting
things, and the CPU's are able to deal with more, more, more.

... JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

fw at deneb

Jul 26, 2008, 3:34 AM

Post #25 of 75 (1779 views)
Permalink

Re: Software router state of the art [In reply to]

* William Herrin:

> The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
> the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.

But they are receive-only, right?

The main problem for "software routing" seems to be that it's basically
Ethernet-only because other interfaces are very difficult to find.

1 2 3

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads