Mailing List Archive: NANOG: users

Hardware capture platforms

1 2

View All

Index | Next | Previous | View Threaded

john at hypergeek

Jul 29, 2008, 4:10 PM

Post #1 of 29 (629 views)
Permalink

Hardware capture platforms

We've deployed a bunch taps in our network and now we need a platform on
which to capture the data. Our bandwidth is currently pretty low but
I've got 8 links to tap, which means I need 16 ports. Has anyone done any
research on doing accurate packet capture with commodity hardware?

--
John A. Kilpatrick
john[at]hypergeek.net Email| http://www.hypergeek.net/
john-page[at]hypergeek.net Text pages| ICQ: 19147504
remember: no obstacles/only challenges

jared at puck

Jul 29, 2008, 4:35 PM

Post #2 of 29 (605 views)
Permalink

Re: Hardware capture platforms [In reply to]

Check out packet forensics depending on what your ultimate
requirements are.

Jared Mauch

On Jul 29, 2008, at 7:10 PM, "John A. Kilpatrick" <john[at]hypergeek.net>
wrote:

>
> We've deployed a bunch taps in our network and now we need a
> platform on which to capture the data. Our bandwidth is currently
> pretty low but I've got 8 links to tap, which means I need 16
> ports. Has anyone done any research on doing accurate packet
> capture with commodity hardware?
>
>
> --
> John A. Kilpatrick
> john[at]hypergeek.net Email| http://www.hypergeek.net/
> john-page[at]hypergeek.net Text pages| ICQ: 19147504
> remember: no obstacles/only challenges
>
>

christian at broknrobot

Jul 29, 2008, 5:11 PM

Post #3 of 29 (604 views)
Permalink

Re: Hardware capture platforms [In reply to]

solera makes some nice boxes also

On Tue, Jul 29, 2008 at 7:35 PM, Jared Mauch <jared[at]puck.nether.net> wrote:

> Check out packet forensics depending on what your ultimate requirements
> are.
>
> Jared Mauch
>
>
> On Jul 29, 2008, at 7:10 PM, "John A. Kilpatrick" <john[at]hypergeek.net>
> wrote:
>
>
>> We've deployed a bunch taps in our network and now we need a platform on
>> which to capture the data. Our bandwidth is currently pretty low but I've
>> got 8 links to tap, which means I need 16 ports. Has anyone done any
>> research on doing accurate packet capture with commodity hardware?
>>
>>
>> --
>> John A. Kilpatrick
>> john[at]hypergeek.net Email| http://www.hypergeek.net/
>> john-page[at]hypergeek.net Text pages| ICQ: 19147504
>> remember: no obstacles/only challenges
>>
>>
>>
>

morrowc.lists at gmail

Jul 29, 2008, 5:12 PM

Post #4 of 29 (605 views)
Permalink

Re: Hardware capture platforms [In reply to]

On Wed, Jul 30, 2008 at 12:35 AM, Jared Mauch <jared[at]puck.nether.net> wrote:
> Check out packet forensics depending on what your ultimate requirements are.
>

I would also add a 'see packet forensics'...

> On Jul 29, 2008, at 7:10 PM, "John A. Kilpatrick" <john[at]hypergeek.net>
> wrote:
>
>>
>> We've deployed a bunch taps in our network and now we need a platform on
>> which to capture the data. Our bandwidth is currently pretty low but I've
>> got 8 links to tap, which means I need 16 ports. Has anyone done any
>> research on doing accurate packet capture with commodity hardware?
>>
>>
>> --
>> John A. Kilpatrick
>> john[at]hypergeek.net Email| http://www.hypergeek.net/
>> john-page[at]hypergeek.net Text pages| ICQ: 19147504
>> remember: no obstacles/only challenges
>>
>>
>
>

netfortius at gmail

Jul 29, 2008, 6:45 PM

Post #5 of 29 (602 views)
Permalink

Re: Hardware capture platforms [In reply to]

Richard's blog @ http://taosecurity.blogspot.com/search?q=taps and
especially his books (Tao of Network Security Monitoring and Extrusion
Detection) are the best sources I have ever found, concerning [not only]
taps and[/but] so much more on the subject - proper usage and best
methodologies and practices for network monitoring (and not only for
security!!!)

Stefan

On Tue, Jul 29, 2008 at 7:12 PM, Christopher Morrow <morrowc.lists[at]gmail.com
> wrote:

> On Wed, Jul 30, 2008 at 12:35 AM, Jared Mauch <jared[at]puck.nether.net>
> wrote:
> > Check out packet forensics depending on what your ultimate requirements
> are.
> >
>
> I would also add a 'see packet forensics'...
>
> > On Jul 29, 2008, at 7:10 PM, "John A. Kilpatrick" <john[at]hypergeek.net>
> > wrote:
> >
> >>
> >> We've deployed a bunch taps in our network and now we need a platform on
> >> which to capture the data. Our bandwidth is currently pretty low but
> I've
> >> got 8 links to tap, which means I need 16 ports. Has anyone done any
> >> research on doing accurate packet capture with commodity hardware?
> >>
> >>
> >> --
> >> John A. Kilpatrick
> >> john[at]hypergeek.net Email| http://www.hypergeek.net/
> >> john-page[at]hypergeek.net Text pages| ICQ: 19147504
> >> remember: no obstacles/only challenges
> >>
> >>
> >
> >
>
>

jpleger at gmail

Jul 29, 2008, 7:26 PM

Post #6 of 29 (601 views)
Permalink

Re: Hardware capture platforms [In reply to]

There are several things that you can do with open source solutions,
however looking at the data may be a bit more difficult than something
like Network Generals or Solera Networks capture appliances. It is
still doable and is definitely much much cheaper...

Something you might want to look into is traffic aggregation with a
switch or hub. You can buy an Allied Telesyn switch and basically turn
it into a hub by disabling switchport learning. Just an idea.

You can use regular old tcpdump with the -C option to rotate logs

tcpdump -i blah -s0 -C <filesize to rotate>, etc.

or you can use Daemonlogger which does pretty much the same thing...

http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html

On Tue, Jul 29, 2008 at 6:45 PM, Network Fortius <netfortius[at]gmail.com> wrote:
> Richard's blog @ http://taosecurity.blogspot.com/search?q=taps and
> especially his books (Tao of Network Security Monitoring and Extrusion
> Detection) are the best sources I have ever found, concerning [not only]
> taps and[/but] so much more on the subject - proper usage and best
> methodologies and practices for network monitoring (and not only for
> security!!!)
>
>
> Stefan
>
> On Tue, Jul 29, 2008 at 7:12 PM, Christopher Morrow <morrowc.lists[at]gmail.com
>> wrote:
>
>> On Wed, Jul 30, 2008 at 12:35 AM, Jared Mauch <jared[at]puck.nether.net>
>> wrote:
>> > Check out packet forensics depending on what your ultimate requirements
>> are.
>> >
>>
>> I would also add a 'see packet forensics'...
>>
>> > On Jul 29, 2008, at 7:10 PM, "John A. Kilpatrick" <john[at]hypergeek.net>
>> > wrote:
>> >
>> >>
>> >> We've deployed a bunch taps in our network and now we need a platform on
>> >> which to capture the data. Our bandwidth is currently pretty low but
>> I've
>> >> got 8 links to tap, which means I need 16 ports. Has anyone done any
>> >> research on doing accurate packet capture with commodity hardware?
>> >>
>> >>
>> >> --
>> >> John A. Kilpatrick
>> >> john[at]hypergeek.net Email| http://www.hypergeek.net/
>> >> john-page[at]hypergeek.net Text pages| ICQ: 19147504
>> >> remember: no obstacles/only challenges
>> >>
>> >>
>> >
>> >
>>
>>
>

ddunkin at netos

Jul 29, 2008, 7:43 PM

Post #7 of 29 (600 views)
Permalink

RE: Hardware capture platforms [In reply to]

Hubs sure are fun...

I would trunk the ports you are monitoring, and run the port monitor on
the trunk port instead (one trunk port, one port per VLAN, plus one
span) which will help with your density. This is assuming the analysis
software you have can read the dot1q tags, but means you do not need to
burn two ports per monitor.

-----Original Message-----
From: James Pleger [mailto:jpleger[at]gmail.com]
Sent: Tuesday, July 29, 2008 19:26
To: nanog[at]merit.edu
Subject: Re: Hardware capture platforms

There are several things that you can do with open source solutions,
however looking at the data may be a bit more difficult than something
like Network Generals or Solera Networks capture appliances. It is
still doable and is definitely much much cheaper...

Something you might want to look into is traffic aggregation with a
switch or hub. You can buy an Allied Telesyn switch and basically turn
it into a hub by disabling switchport learning. Just an idea.

You can use regular old tcpdump with the -C option to rotate logs

tcpdump -i blah -s0 -C <filesize to rotate>, etc.

or you can use Daemonlogger which does pretty much the same thing...

http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html

On Tue, Jul 29, 2008 at 6:45 PM, Network Fortius <netfortius[at]gmail.com>
wrote:
> Richard's blog @ http://taosecurity.blogspot.com/search?q=taps and
> especially his books (Tao of Network Security Monitoring and Extrusion
> Detection) are the best sources I have ever found, concerning [not
only]
> taps and[/but] so much more on the subject - proper usage and best
> methodologies and practices for network monitoring (and not only for
> security!!!)
>
>
> Stefan
>
> On Tue, Jul 29, 2008 at 7:12 PM, Christopher Morrow
<morrowc.lists[at]gmail.com
>> wrote:
>
>> On Wed, Jul 30, 2008 at 12:35 AM, Jared Mauch <jared[at]puck.nether.net>
>> wrote:
>> > Check out packet forensics depending on what your ultimate
requirements
>> are.
>> >
>>
>> I would also add a 'see packet forensics'...
>>
>> > On Jul 29, 2008, at 7:10 PM, "John A. Kilpatrick"
<john[at]hypergeek.net>
>> > wrote:
>> >
>> >>
>> >> We've deployed a bunch taps in our network and now we need a
platform on
>> >> which to capture the data. Our bandwidth is currently pretty low
but
>> I've
>> >> got 8 links to tap, which means I need 16 ports. Has anyone done
any
>> >> research on doing accurate packet capture with commodity hardware?
>> >>
>> >>
>> >> --
>> >> John A. Kilpatrick
>> >> john[at]hypergeek.net Email|
http://www.hypergeek.net/
>> >> john-page[at]hypergeek.net Text pages| ICQ: 19147504
>> >> remember: no obstacles/only challenges
>> >>
>> >>
>> >
>> >
>>
>>
>

mkarir at merit

Jul 29, 2008, 7:54 PM

Post #8 of 29 (601 views)
Permalink

Re: Hardware capture platforms [In reply to]

Hi John,

You might want to check out www.opencalea.org. We have just
released opencalea-lite which is a complete re-write of the original
opencalea software. OpenCalea-lite is a much better and cleaner
re-write(we learnt from our mistakes in the previous releases).
One of the problems of the original version was that we were
getting bogged down in details over the precise standard format
instead of making the core more stable.
OpenCalea-lite takes a step back form this and aims at
doing well the essense of what packet taps should be able to.
It has a nice clean tap/controller/collector architecture which is much
more robust. Taps will register with the controller irrespective of
which is started first. Process control has also been improved.
Starting and stopping taps is handled in a much cleaner way.
In addtion TCP streams are used to transfer data.
We were about to send out an announcement
regarding opencalea-lite on the opencalea[at]merit.edu
mailing list. Aside from calea requirements opencalea-lite is
actually a fairly good platform for running remote-taps in
your network.

-manish

> Message: 4
> Date: Tue, 29 Jul 2008 16:10:09 -0700 (PDT)
> From: "John A. Kilpatrick" <john[at]hypergeek.net>
> Subject: Hardware capture platforms
> To: nanog[at]merit.edu
> Message-ID: <20080729155511.R42026[at]iama.hypergeek.net>
> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>
>
> We've deployed a bunch taps in our network and now we need a platform
> on
> which to capture the data. Our bandwidth is currently pretty low but
> I've got 8 links to tap, which means I need 16 ports. Has anyone done
> any
> research on doing accurate packet capture with commodity hardware?
>
>
> --
> John A. Kilpatrick
> john[at]hypergeek.net Email| http://www.hypergeek.net/
> john-page[at]hypergeek.net Text pages| ICQ: 19147504
> remember: no obstacles/only challenges
>

seclists at rm-rf

Jul 30, 2008, 6:26 AM

Post #9 of 29 (588 views)
Permalink

Re: Hardware capture platforms [In reply to]

On 30 Jul 2008, at 03:26, James Pleger wrote:
>
> Something you might want to look into is traffic aggregation with a
> switch or hub. You can buy an Allied Telesyn switch and basically turn
> it into a hub by disabling switchport learning. Just an idea.

Never try to aggregate multiple TAPs with a hub.
You will just create a bucket load of collisions and end up with a
useless data feed presented to your monitoring tool. If you want to
aggregate multiple TAP feeds into a smaller number of devices(s), most
of the TAP vendors make some form of link aggregation device.

Or, depending on the OS and sniffer you use, you may be able to bond
the interfaces on the capture device.

-Leon

>
> You can use regular old tcpdump with the -C option to rotate logs
>
> tcpdump -i blah -s0 -C <filesize to rotate>, etc.
>
> or you can use Daemonlogger which does pretty much the same thing...
>
> http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html

warren at kumari

Jul 30, 2008, 11:32 AM

Post #10 of 29 (585 views)
Permalink

Re: Hardware capture platforms [In reply to]

On Jul 29, 2008, at 10:43 PM, Darryl Dunkin wrote:

> Hubs sure are fun...
>

This might be a stupid question, but where can one get small hubs
these days? All of the common commodity (eg: 4 port Netgear) "hubs"
these days are actually switches.

What I am looking for is:
Small enough to live in my notebook bag (e.g.: 4 port with a wall wart.)
Cheap
Simple
10/100/1000Mbps

While a tap would work, I'd prefer a hub because I can then use it to
connect machines together in a pinch.

W
---

In the past I have bought some cheap 4 port commodity switches (form
Circuit City or somewhere similar), found the datasheet for the
chipset (it was a Broadcom something or other) and tied the pin to
ground that disables the learning mode (actually, I think that the pin
just set the size of the learning table to be 0 entries). While this
works, doing it once was more than enough :-)

> I would trunk the ports you are monitoring, and run the port monitor
> on
> the trunk port instead (one trunk port, one port per VLAN, plus one
> span) which will help with your density. This is assuming the analysis
> software you have can read the dot1q tags, but means you do not need
> to
> burn two ports per monitor.
>
> -----Original Message-----
> From: James Pleger [mailto:jpleger[at]gmail.com]
> Sent: Tuesday, July 29, 2008 19:26
> To: nanog[at]merit.edu
> Subject: Re: Hardware capture platforms
>
> There are several things that you can do with open source solutions,
> however looking at the data may be a bit more difficult than something
> like Network Generals or Solera Networks capture appliances. It is
> still doable and is definitely much much cheaper...
>
> Something you might want to look into is traffic aggregation with a
> switch or hub. You can buy an Allied Telesyn switch and basically turn
> it into a hub by disabling switchport learning. Just an idea.
>
> You can use regular old tcpdump with the -C option to rotate logs
>
> tcpdump -i blah -s0 -C <filesize to rotate>, etc.
>
> or you can use Daemonlogger which does pretty much the same thing...
>
> http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html
>
>
> On Tue, Jul 29, 2008 at 6:45 PM, Network Fortius
> <netfortius[at]gmail.com>
> wrote:
>> Richard's blog @ http://taosecurity.blogspot.com/search?q=taps and
>> especially his books (Tao of Network Security Monitoring and
>> Extrusion
>> Detection) are the best sources I have ever found, concerning [not
> only]
>> taps and[/but] so much more on the subject - proper usage and best
>> methodologies and practices for network monitoring (and not only for
>> security!!!)
>>
>>
>> Stefan
>>
>> On Tue, Jul 29, 2008 at 7:12 PM, Christopher Morrow
> <morrowc.lists[at]gmail.com
>>> wrote:
>>
>>> On Wed, Jul 30, 2008 at 12:35 AM, Jared Mauch
>>> <jared[at]puck.nether.net>
>>> wrote:
>>>> Check out packet forensics depending on what your ultimate
> requirements
>>> are.
>>>>
>>>
>>> I would also add a 'see packet forensics'...
>>>
>>>> On Jul 29, 2008, at 7:10 PM, "John A. Kilpatrick"
> <john[at]hypergeek.net>
>>>> wrote:
>>>>
>>>>>
>>>>> We've deployed a bunch taps in our network and now we need a
> platform on
>>>>> which to capture the data. Our bandwidth is currently pretty low
> but
>>> I've
>>>>> got 8 links to tap, which means I need 16 ports. Has anyone done
> any
>>>>> research on doing accurate packet capture with commodity hardware?
>>>>>
>>>>>
>>>>> --
>>>>> John A. Kilpatrick
>>>>> john[at]hypergeek.net Email|
> http://www.hypergeek.net/
>>>>> john-page[at]hypergeek.net Text pages| ICQ: 19147504
>>>>> remember: no obstacles/only challenges
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
>

--
"Build a man a fire, and he'll be warm for a day. Set a man on fire,
and he'll be warm for the rest of his life." -- Terry Pratchett

Jon.Kibler at aset

Jul 30, 2008, 11:47 AM

Post #11 of 29 (585 views)
Permalink

Re: Hardware capture platforms [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Warren Kumari wrote:
>
> On Jul 29, 2008, at 10:43 PM, Darryl Dunkin wrote:
>
>> Hubs sure are fun...
>>
>
> This might be a stupid question, but where can one get small hubs these
> days? All of the common commodity (eg: 4 port Netgear) "hubs" these
> days are actually switches.
>
> What I am looking for is:
> Small enough to live in my notebook bag (e.g.: 4 port with a wall wart.)
> Cheap
> Simple
> 10/100/1000Mbps
>
> While a tap would work, I'd prefer a hub because I can then use it to
> connect machines together in a pinch.
>

Hubs are still available that are REAL hubs. I got 4 netgears about a
year ago and they are still available.

However, there is a problem with your specification: No hub (that I am
aware of) can do 1Gbps. All hubs are 10/100 AFAIK.

Jon Kibler
- --
Jon R. Kibler
Chief Technical Officer
Advanced Systems Engineering Technology, Inc.
Charleston, SC USA
o: 843-849-8214
c: 843-224-2494
s: 843-564-4224

My PGP Fingerprint is:
BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkiQty8ACgkQUVxQRc85QlOA1ACfWWGa6FcwzcKT1PN+0pBRky46
bUQAnAxgqV4hfGEZBSgPoMXP8+3/PS+k
=ynxx
-----END PGP SIGNATURE-----

==================================================
Filtered by: TRUSTEM.COM's Email Filtering Service
http://www.trustem.com/
No Spam. No Viruses. Just Good Clean Email.

shrdlu at deaddrop

Jul 30, 2008, 11:52 AM

Post #12 of 29 (585 views)
Permalink

Re: Hardware capture platforms [In reply to]

Warren Kumari wrote:

>
> On Jul 29, 2008, at 10:43 PM, Darryl Dunkin wrote:
>
>> Hubs sure are fun...

> This might be a stupid question, but where can one get small hubs these
> days? All of the common commodity (eg: 4 port Netgear) "hubs" these
> days are actually switches.

True enough. For those of us who need and want something non-switched,
eBay and other used hardware places are the only real option.

> What I am looking for is: Small enough to live in my notebook bag
> (e.g.: 4 port with a wall wart.) Cheap Simple 10/100/1000Mbps

I don't believe that such a thing ever existed. Hubs that did 10/100,
certainly, but I've never ever seen a hub that did gig speeds. When I
realized hubs were about to be an endangered species, I started
purchasing new and used. I have at least two that (other than testing)
have never been used.

> While a tap would work, I'd prefer a hub because I can then use it to
> connect machines together in a pinch.

The original poster needed to deploy a tap, and a hub (for him) would
defeat the purpose entirely. If you really really need a hub (or two),
your best bet is to start looking at various resellers. Pity you're not
closer; I'm retired, and no longer really need the six or eight that I
still have.

--
In April 1951, Galaxy published C.M. Kornbluth's "The Marching Morons".
The intervening years have proven Kornbluth right.
--Valdis Kletnieks

mhuff at ox

Jul 30, 2008, 12:09 PM

Post #13 of 29 (587 views)
Permalink

RE: Hardware capture platforms [In reply to]

The Cisco 8 port 10/100/1000 switch (WS-C2960G-8TC-L) supports RSPAN which would allow you to tap all the ports even though it's a switch. It's about $750, so it's not a cheap option, but it's not outrageous either. It's the right size also.

----
Matthew Huff | One Manhattanville Rd
OTA Management LLC | Purchase, NY 10577
www.otaotr.com | Phone: 914-460-4039
aim: matthewbhuff | Fax: 914-460-4139
-----Original Message-----
From: Lynda [mailto:shrdlu[at]deaddrop.org]
Sent: Wednesday, July 30, 2008 2:52 PM
To: Nanog
Subject: Re: Hardware capture platforms

Warren Kumari wrote:

>
> On Jul 29, 2008, at 10:43 PM, Darryl Dunkin wrote:
>
>> Hubs sure are fun...

> This might be a stupid question, but where can one get small hubs
> these days? All of the common commodity (eg: 4 port Netgear) "hubs"
> these days are actually switches.

True enough. For those of us who need and want something non-switched, eBay and other used hardware places are the only real option.

> What I am looking for is: Small enough to live in my notebook bag
> (e.g.: 4 port with a wall wart.) Cheap Simple 10/100/1000Mbps

I don't believe that such a thing ever existed. Hubs that did 10/100, certainly, but I've never ever seen a hub that did gig speeds. When I realized hubs were about to be an endangered species, I started purchasing new and used. I have at least two that (other than testing) have never been used.

> While a tap would work, I'd prefer a hub because I can then use it to
> connect machines together in a pinch.

The original poster needed to deploy a tap, and a hub (for him) would defeat the purpose entirely. If you really really need a hub (or two), your best bet is to start looking at various resellers. Pity you're not closer; I'm retired, and no longer really need the six or eight that I still have.

--
In April 1951, Galaxy published C.M. Kornbluth's "The Marching Morons".
The intervening years have proven Kornbluth right.
--Valdis Kletnieks

nathan at robotics

Jul 30, 2008, 3:44 PM

Post #14 of 29 (584 views)
Permalink

Re: Hardware capture platforms [In reply to]

On Tue, 29 Jul 2008, John A. Kilpatrick wrote:

> We've deployed a bunch taps in our network and now we need a platform on
> which to capture the data. Our bandwidth is currently pretty low but I've
> got 8 links to tap, which means I need 16 ports. Has anyone done any
> research on doing accurate packet capture with commodity hardware?

A hardware based capture card is the only way to get to any real
throughput. Check out Endace cards, that will let you do line rate gig e
or better and has native libpcap interface. You also may want to check out
WildPackets cards.

><>
Nathan Stratton CTO, BlinkMind, Inc.
nathan at robotics.net nathan at blinkmind.com
http://www.robotics.net http://www.blinkmind.com

dnewman at networktest

Jul 30, 2008, 4:34 PM

Post #15 of 29 (570 views)
Permalink

Re: Hardware capture platforms [In reply to]

Jon Kibler wrote:

> Hubs are still available that are REAL hubs. I got 4 netgears about a
> year ago and they are still available.
>
> However, there is a problem with your specification: No hub (that I am
> aware of) can do 1Gbps. All hubs are 10/100 AFAIK.

Grand Junction made a gigabit Ethernet repeater around 1996. It was
based on the "carrier extension" part of the gigabit Ethernet
spec that allows for half-duplex operation. Carrier extension pads any
frame shorter than 512 bytes to be 512 bytes long.

For that reason (in case frame size distribution matters), as well as
the tons of collisions that others have mentioned, I'd also stay away
from hubs for the OP's needs.

Also, many 10/100 hubs have a 2-port switch to move frames between
speeds, so it's conceivable that even a "hub" may have multiple
collision domains.

dn

rdobbins at cisco

Jul 30, 2008, 5:10 PM

Post #16 of 29 (570 views)
Permalink

Re: Hardware capture platforms [In reply to]

On Jul 31, 2008, at 5:44 AM, nathan[at]robotics.net wrote:

> Check out Endace cards, that will let you do line rate gig e or
> better and has native libpcap interface.

I believe Endace also have a productized box containing their capture
cards (NinjaProbe); it can be used to capture packets, and can also
export NetFlow telemetry based upon the captured traffic. Arbor,
Narus, and Lancope have similar NetFlow-via-packet-capture capabilities.

-----------------------------------------------------------------------
Roland Dobbins <rdobbins[at]cisco.com> // +66.83.266.6344 mobile

History is a great teacher, but it also lies with impunity.

-- John Robb

ljb at merit

Jul 30, 2008, 10:50 PM

Post #17 of 29 (549 views)
Permalink

Re: Hardware capture platforms [In reply to]

Warren Kumari wrote:
>
> On Jul 29, 2008, at 10:43 PM, Darryl Dunkin wrote:
>
>> Hubs sure are fun...
>>
>
> This might be a stupid question, but where can one get small hubs
> these days? All of the common commodity (eg: 4 port Netgear) "hubs"
> these days are actually switches.
>
> What I am looking for is:
> Small enough to live in my notebook bag (e.g.: 4 port with a wall wart.)
> Cheap
> Simple
> 10/100/1000Mbps
>
> While a tap would work, I'd prefer a hub because I can then use it to
> connect machines together in a pinch.

D-Link sells a smallish 8-port managed Gigabit switch that allows
you to disable learning on the ports -- DGS-3200-10 --
http://www.dlink.com/products/?sec=0&pid=674
I don't know where they hide the manuals on the D-Link
US site, but Google turned them up on their Russian ftp server ??
While not incredibly cheap, it seems reasonable at about $300.
As a bonus, it seems to have pretty complete IPv6 support.

We wanted to do something similar with a 10G switch (SMC8708L2).
It let's you set the size of the MAC table, but not to zero. However,
we found that setting the size of the table to 1 entry effectively disabled
learning.

>
> W
> ---
>
> In the past I have bought some cheap 4 port commodity switches (form
> Circuit City or somewhere similar), found the datasheet for the
> chipset (it was a Broadcom something or other) and tied the pin to
> ground that disables the learning mode (actually, I think that the pin
> just set the size of the learning table to be 0 entries). While this
> works, doing it once was more than enough :-)
>
Nice hack!

sam_mailinglists at spacething

Jul 31, 2008, 1:53 AM

Post #18 of 29 (547 views)
Permalink

Re: Hardware capture platforms [In reply to]

Lynda wrote:
> Warren Kumari wrote:
>
>> What I am looking for is: Small enough to live in my notebook bag
>> (e.g.: 4 port with a wall wart.) Cheap Simple 10/100/1000Mbps
>
> I don't believe that such a thing ever existed. Hubs that did 10/100,
> certainly, but I've never ever seen a hub that did gig speeds.
>
Depends what you mean by 'hub' I guess. I thought the term referred to a
device that was half-duplex only, and had no address learning. GE has
never supported half-duplex.

Sam

joelja at bogus

Jul 31, 2008, 2:04 AM

Post #19 of 29 (548 views)
Permalink

Re: Hardware capture platforms [In reply to]

Warren Kumari wrote:
>
> On Jul 29, 2008, at 10:43 PM, Darryl Dunkin wrote:
>
>> Hubs sure are fun...
>>
>
> This might be a stupid question, but where can one get small hubs these
> days? All of the common commodity (eg: 4 port Netgear) "hubs" these
> days are actually switches.
>
> What I am looking for is:
> Small enough to live in my notebook bag (e.g.: 4 port with a wall wart.)
> Cheap
> Simple
> 10/100/1000Mbps

You won't find the gig-e hub out there for sale despite some ieee 802.3
participants staunch defense of 1/2 duplex gig-e support and the
resulting complications that caused/s...

Perversely when traveling I actually use the Ethernet ports on my
soekris configured as a bridge for this application. A device with 4
Ethernet ports plus a wifi radio which can be configured as bridges,
routed, nated etc if that's what's desired. the soekris is not gig-e
capable and it's forwarding capacity is a bit closer to the low hundreds
of megs, but it travels in my bag, has disk, wifi etc.

MSI industrial makes a mini-itx mainboard that will take an intel core2
has 3 embedded gig-e ports and a 16x pci-e slot that you can put a
multiport gig or 2 x 10Gbe interface in... I have a utility 10" deep
rackmount that I drag around with that in it when I need more power than
the soekris can deliver...

http://www.logicsupply.com/products/ms_9642

> While a tap would work, I'd prefer a hub because I can then use it to
> connect machines together in a pinch.
>
> W
> ---
>
> In the past I have bought some cheap 4 port commodity switches (form
> Circuit City or somewhere similar), found the datasheet for the chipset
> (it was a Broadcom something or other) and tied the pin to ground that
> disables the learning mode (actually, I think that the pin just set the
> size of the learning table to be 0 entries). While this works, doing it
> once was more than enough :-)
>
>> I would trunk the ports you are monitoring, and run the port monitor on
>> the trunk port instead (one trunk port, one port per VLAN, plus one
>> span) which will help with your density. This is assuming the analysis
>> software you have can read the dot1q tags, but means you do not need to
>> burn two ports per monitor.
>>
>> -----Original Message-----
>> From: James Pleger [mailto:jpleger[at]gmail.com]
>> Sent: Tuesday, July 29, 2008 19:26
>> To: nanog[at]merit.edu
>> Subject: Re: Hardware capture platforms
>>
>> There are several things that you can do with open source solutions,
>> however looking at the data may be a bit more difficult than something
>> like Network Generals or Solera Networks capture appliances. It is
>> still doable and is definitely much much cheaper...
>>
>> Something you might want to look into is traffic aggregation with a
>> switch or hub. You can buy an Allied Telesyn switch and basically turn
>> it into a hub by disabling switchport learning. Just an idea.
>>
>> You can use regular old tcpdump with the -C option to rotate logs
>>
>> tcpdump -i blah -s0 -C <filesize to rotate>, etc.
>>
>> or you can use Daemonlogger which does pretty much the same thing...
>>
>> http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html
>>
>>
>> On Tue, Jul 29, 2008 at 6:45 PM, Network Fortius <netfortius[at]gmail.com>
>> wrote:
>>> Richard's blog @ http://taosecurity.blogspot.com/search?q=taps and
>>> especially his books (Tao of Network Security Monitoring and Extrusion
>>> Detection) are the best sources I have ever found, concerning [not
>> only]
>>> taps and[/but] so much more on the subject - proper usage and best
>>> methodologies and practices for network monitoring (and not only for
>>> security!!!)
>>>
>>>
>>> Stefan
>>>
>>> On Tue, Jul 29, 2008 at 7:12 PM, Christopher Morrow
>> <morrowc.lists[at]gmail.com
>>>> wrote:
>>>
>>>> On Wed, Jul 30, 2008 at 12:35 AM, Jared Mauch <jared[at]puck.nether.net>
>>>> wrote:
>>>>> Check out packet forensics depending on what your ultimate
>> requirements
>>>> are.
>>>>>
>>>>
>>>> I would also add a 'see packet forensics'...
>>>>
>>>>> On Jul 29, 2008, at 7:10 PM, "John A. Kilpatrick"
>> <john[at]hypergeek.net>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> We've deployed a bunch taps in our network and now we need a
>> platform on
>>>>>> which to capture the data. Our bandwidth is currently pretty low
>> but
>>>> I've
>>>>>> got 8 links to tap, which means I need 16 ports. Has anyone done
>> any
>>>>>> research on doing accurate packet capture with commodity hardware?
>>>>>>
>>>>>>
>>>>>> --
>>>>>> John A. Kilpatrick
>>>>>> john[at]hypergeek.net Email|
>> http://www.hypergeek.net/
>>>>>> john-page[at]hypergeek.net Text pages| ICQ: 19147504
>>>>>> remember: no obstacles/only challenges
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
> --
> "Build a man a fire, and he'll be warm for a day. Set a man on fire, and
> he'll be warm for the rest of his life." -- Terry Pratchett
>
>
>

juuso.lehtinen at gmail

Jul 31, 2008, 6:16 AM

Post #20 of 29 (539 views)
Permalink

Re: Hardware capture platforms [In reply to]

Second that.

Using hub to tap into a single link is also risky. I used to monitor single
FE link with 100M hub. After link had moderate utilization >20%, collision
led was lit all the time.

I've had good experience with VSS Monitoring Ethernet Aggregator taps. Also
Catalyst 2960 SPAN seems to work OK.

As for capture PC, we've been using regular PC with Wireshark. That's good
for single FE link, but has problem with GE and multiple links.

BR,
Juuso

On Wed, Jul 30, 2008 at 4:26 PM, Leon Ward <seclists[at]rm-rf.co.uk> wrote:

>
> On 30 Jul 2008, at 03:26, James Pleger wrote:
>
>>
>> Something you might want to look into is traffic aggregation with a
>> switch or hub. You can buy an Allied Telesyn switch and basically turn
>> it into a hub by disabling switchport learning. Just an idea.
>>
>
> Never try to aggregate multiple TAPs with a hub.
> You will just create a bucket load of collisions and end up with a useless
> data feed presented to your monitoring tool. If you want to aggregate
> multiple TAP feeds into a smaller number of devices(s), most of the TAP
> vendors make some form of link aggregation device.
>
> Or, depending on the OS and sniffer you use, you may be able to bond the
> interfaces on the capture device.
>
> -Leon
>
>
>
>
>> You can use regular old tcpdump with the -C option to rotate logs
>>
>> tcpdump -i blah -s0 -C <filesize to rotate>, etc.
>>
>> or you can use Daemonlogger which does pretty much the same thing...
>>
>> http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html
>>
>
>
>

seclists at rm-rf

Jul 31, 2008, 8:00 AM

Post #21 of 29 (538 views)
Permalink

Re: Hardware capture platforms [In reply to]

On 31 Jul 2008, at 14:16, Juuso Lehtinen wrote:

> Second that.
>
> Using hub to tap into a single link is also risky. I used to monitor
> single FE link with 100M hub. After link had moderate utilization
> >20%, collision led was lit all the time.
>
> I've had good experience with VSS Monitoring Ethernet Aggregator
> taps. Also Catalyst 2960 SPAN seems to work OK.
>
> As for capture PC, we've been using regular PC with Wireshark.
> That's good for single FE link, but has problem with GE and multiple
> links.

If you need to increase the speed of your capture tool, maybe this [1]
link may be of use.
It is an implementation of a libpcap that implements a shared memory
ring buffer which can result in some capture performance gains.

[1] http://public.lanl.gov/cpw/

-Leon

> BR,
> Juuso
>
> On Wed, Jul 30, 2008 at 4:26 PM, Leon Ward <seclists[at]rm-rf.co.uk>
> wrote:
>
> On 30 Jul 2008, at 03:26, James Pleger wrote:
>
> Something you might want to look into is traffic aggregation with a
> switch or hub. You can buy an Allied Telesyn switch and basically turn
> it into a hub by disabling switchport learning. Just an idea.
>
> Never try to aggregate multiple TAPs with a hub.
> You will just create a bucket load of collisions and end up with a
> useless data feed presented to your monitoring tool. If you want to
> aggregate multiple TAP feeds into a smaller number of devices(s),
> most of the TAP vendors make some form of link aggregation device.
>
> Or, depending on the OS and sniffer you use, you may be able to bond
> the interfaces on the capture device.
>
> -Leon
>
>
>
>
> You can use regular old tcpdump with the -C option to rotate logs
>
> tcpdump -i blah -s0 -C <filesize to rotate>, etc.
>
> or you can use Daemonlogger which does pretty much the same thing...
>
> http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html
>
>
>

jra at baylink

Jul 31, 2008, 9:31 AM

Post #22 of 29 (535 views)
Permalink

Re: Hardware capture platforms [In reply to]

On Wed, Jul 30, 2008 at 02:47:11PM -0400, Jon Kibler wrote:
> Hubs are still available that are REAL hubs. I got 4 netgears about a
> year ago and they are still available.
>
> However, there is a problem with your specification: No hub (that I am
> aware of) can do 1Gbps. All hubs are 10/100 AFAIK.

And, note carefully: some "dual-speed hubs" are actually a 10BT hub and
a 100BT hub *with a switch between them*. I forget which brand I
caught this on, but it bit me a couple of years back.

Which speed cable you plug in determines which hub you're talking to.

Yes, it's weird.

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra[at]baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com '87 e24
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

Those who cast the vote decide nothing.
Those who count the vote decide everything.
-- (Josef Stalin)

warren at kumari

Jul 31, 2008, 10:04 AM

Post #23 of 29 (536 views)
Permalink

Re: Hardware capture platforms [In reply to]

On Jul 31, 2008, at 12:31 PM, Jay R. Ashworth wrote:

> On Wed, Jul 30, 2008 at 02:47:11PM -0400, Jon Kibler wrote:
>> Hubs are still available that are REAL hubs. I got 4 netgears about a
>> year ago and they are still available.
>>
>> However, there is a problem with your specification: No hub (that I
>> am
>> aware of) can do 1Gbps. All hubs are 10/100 AFAIK.

Ok, so I guess what I am speaking is not strictly a hub, it is a non-
learning bridge (single collision domain per port, full duplex, etc).
There used to be a bunch of devices sold like this -- there were a few
really cheap chipsets (AFAIR, Vitesse SparX VSCsomething was one of
them -- basically a standard switch chipset that they shaved a few
cents off because there was no learning logic / memory) that many
people used in cheap "hubs"... I still have some of these somewhere
and will rip the lid off to figure out exactly what it was so I can
get some more...

>>
>
> And, note carefully: some "dual-speed hubs" are actually a 10BT hub
> and
> a 100BT hub *with a switch between them*. I forget which brand I
> caught this on, but it bit me a couple of years back.
>
> Which speed cable you plug in determines which hub you're talking to.

I see your weird hub story and raise you one:

I went along to one of my wife's clients to help lug a printer up the
stairs... We get it on the desk and I go to plug in the Ethernet port
-- I follow some cables and find this small white switch jammed behind
a photocopier -- I pull it out and it has, emblazoned in large red
letters on the front, "10/100 Hub with Switch" -- this was back in
the day when switches were still cool... I turn it around, and on the
back there is... a switch, one side is marked "10M" and the other is
marked "100M"... After I stopped laughing I tested it, and sure
enough, its a standard hub, and you can make the ports either run at
10Mbps or 100Mbps by flipping the switch... I *really* wish I had
replaced and kept it...

W
>
>
> Yes, it's weird.
>
> Cheers,
> -- jra
> --
> Jay R. Ashworth Baylink jra[at]baylink.com
> Designer The Things I
> Think RFC 2100
> Ashworth & Associates http://
> baylink.pitas.com '87 e24
> St Petersburg FL USA http://photo.imageinc.us +1
> 727 647 1274
>
> Those who cast the vote decide nothing.
> Those who count the vote decide everything.
> -- (Josef Stalin)
>

--
Do not meddle in the affairs of wizards, for they are subtle and quick
to anger.
-- J.R.R. Tolkien

meekjt at gmail

Jul 31, 2008, 11:46 AM

Post #24 of 29 (530 views)
Permalink

Re: Hardware capture platforms [In reply to]

I have had the same problem and solved it with a rare (even then)
100BT Only hub. I still have at least one stashed away.

For years though, I have been using bonding on Linux to combine multiple
tap streams. We also use hardware aggregators for the higher volume
applications.

Jon

On Thu, Jul 31, 2008 at 12:31 PM, Jay R. Ashworth <jra[at]baylink.com> wrote:
>
> And, note carefully: some "dual-speed hubs" are actually a 10BT hub and
> a 100BT hub *with a switch between them*. I forget which brand I
> caught this on, but it bit me a couple of years back.
>
> Which speed cable you plug in determines which hub you're talking to.
>
> Yes, it's weird.
>
> Cheers,
> -- jra
> --
> Jay R. Ashworth Baylink jra[at]baylink.com
> Designer The Things I Think RFC 2100
> Ashworth & Associates http://baylink.pitas.com '87 e24
> St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
>
> Those who cast the vote decide nothing.
> Those who count the vote decide everything.
> -- (Josef Stalin)
>
>

nikky at mnet

Jul 31, 2008, 12:37 PM

Post #25 of 29 (527 views)
Permalink

Re: Hardware capture platforms [In reply to]

Hey,

On Thu, 31 Jul 2008 16:00:36 +0100
Leon Ward <seclists[at]rm-rf.co.uk> wrote:

>
> On 31 Jul 2008, at 14:16, Juuso Lehtinen wrote:
>
> > Second that.
> >
> > Using hub to tap into a single link is also risky. I used to monitor
> > single FE link with 100M hub. After link had moderate utilization
> > >20%, collision led was lit all the time.
> >
> > I've had good experience with VSS Monitoring Ethernet Aggregator
> > taps. Also Catalyst 2960 SPAN seems to work OK.
> >
> > As for capture PC, we've been using regular PC with Wireshark.
> > That's good for single FE link, but has problem with GE and multiple
> > links.
>
> If you need to increase the speed of your capture tool, maybe this [1]
> link may be of use.
> It is an implementation of a libpcap that implements a shared memory
> ring buffer which can result in some capture performance gains.
>
> [1] http://public.lanl.gov/cpw/

Better off - http://www.ntop.org/PF_RING.html
I've seen tenfold decrease in CPU usage using PF_RING.

>
> -Leon

[ cut ]

--
Best regards,
Nickola Kolev

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com