Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NANOG: users

Network Traffic Collection

 

 

NANOG users RSS feed   Index | Next | Previous | View Threaded


myeaddress at gmail

Feb 23, 2012, 12:11 PM

Post #1 of 17 (887 views)
Permalink
Network Traffic Collection

Hello,

I am trying to collect traffic traffic from pcap file and store it in
a database but really confused how to organize it. Should I organize
it on connection basis/ flow basis or IP basis.

It might be an effort to write a customized traffic analysis tool like
wireshark with only required functionality. I would really appreciate
if someone can give me direction on write way of organizing the data
because right now I only see individual packets and no way of putting
them in some order.

Best,
Ali


jeroen at unfix

Feb 23, 2012, 12:14 PM

Post #2 of 17 (860 views)
Permalink
Re: Network Traffic Collection [In reply to]

On 2012-02-23 21:11 , Maverick wrote:
> Hello,
>
> I am trying to collect traffic traffic from pcap file and store it in
> a database but really confused how to organize it. Should I organize
> it on connection basis/ flow basis or IP basis.
>
> It might be an effort to write a customized traffic analysis tool like
> wireshark with only required functionality. I would really appreciate
> if someone can give me direction on write way of organizing the data
> because right now I only see individual packets and no way of putting
> them in some order.

Does this all not completely depend on what you actually want to do with
it? You might want to start there instead of the other way around.

Greets,
Jeroen


myeaddress at gmail

Feb 23, 2012, 12:19 PM

Post #3 of 17 (867 views)
Permalink
Re: Network Traffic Collection [In reply to]

I want to be able to see information like how much traffic an ip send
over a period of time, what machines it talked to etc from this
perspective it should be IP based but I would really like to know how
other people do it.

Best,
Ali

On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen [at] unfix> wrote:
> On 2012-02-23 21:11 , Maverick wrote:
>> Hello,
>>
>> I am trying to collect traffic traffic from pcap file and store it in
>> a database but really confused how to organize it. Should I organize
>> it on connection basis/ flow basis or IP basis.
>>
>> It might be an effort to write a customized traffic analysis tool like
>> wireshark with only required functionality. I would really appreciate
>> if someone can give me direction on write way of organizing the data
>> because right now I only see individual packets and no way of putting
>> them in some order.
>
> Does this all not completely depend on what you actually want to do with
> it? You might want to start there instead of the other way around.
>
> Greets,
> Jeroen
>


MatlockK at exempla

Feb 23, 2012, 12:20 PM

Post #4 of 17 (859 views)
Permalink
RE: Network Traffic Collection [In reply to]

Netflow + netflow collector.

Ken Matlock
Network Analyst
Systems and Technology Service Center
Sisters of Charity of Leavenworth Health System
12600 W. Colfax, Suite A-500
Lakewood, CO 80215

303-467-4671
matlockk [at] exempla

-----Original Message-----
From: Maverick [mailto:myeaddress [at] gmail]
Sent: Thursday, February 23, 2012 1:19 PM
To: Jeroen Massar
Cc: nanog [at] nanog
Subject: Re: Network Traffic Collection

I want to be able to see information like how much traffic an ip send over a period of time, what machines it talked to etc from this perspective it should be IP based but I would really like to know how other people do it.

Best,
Ali

On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen [at] unfix> wrote:
> On 2012-02-23 21:11 , Maverick wrote:
>> Hello,
>>
>> I am trying to collect traffic traffic from pcap file and store it in
>> a database but really confused how to organize it. Should I organize
>> it on connection basis/ flow basis or IP basis.
>>
>> It might be an effort to write a customized traffic analysis tool
>> like wireshark with only required functionality. I would really
>> appreciate if someone can give me direction on write way of
>> organizing the data because right now I only see individual packets
>> and no way of putting them in some order.
>
> Does this all not completely depend on what you actually want to do
> with it? You might want to start there instead of the other way around.
>
> Greets,
> Jeroen
>

*** Exempla Confidentiality Notice *** The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any other dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify me immediately by replying to the message and deleting it from your computer. Thank you. *** Exempla Confidentiality Notice ***


sraja97 at gmail

Feb 23, 2012, 12:29 PM

Post #5 of 17 (865 views)
Permalink
Re: Network Traffic Collection [In reply to]

On Thu, Feb 23, 2012 at 12:19 PM, Maverick <myeaddress [at] gmail> wrote:
> I want to be able to see information like how much traffic an ip send
> over a period of time, what machines it talked to etc from this
> perspective it should be IP based but I would really like to know how
> other people do it.
>


Run argus on a span port.

-Suresh


mike.lyon at gmail

Feb 23, 2012, 12:34 PM

Post #6 of 17 (859 views)
Permalink
Re: Network Traffic Collection [In reply to]

Random thought, anyone ever used Splunk for this kind of thing?

-mike

Sent from my iPhone

On Feb 23, 2012, at 10:30, Suresh Rajagopalan <sraja97 [at] gmail> wrote:

> On Thu, Feb 23, 2012 at 12:19 PM, Maverick <myeaddress [at] gmail> wrote:
>> I want to be able to see information like how much traffic an ip send
>> over a period of time, what machines it talked to etc from this
>> perspective it should be IP based but I would really like to know how
>> other people do it.
>>
>
>
> Run argus on a span port.
>
> -Suresh
>


jason at lixfeld

Feb 23, 2012, 12:51 PM

Post #7 of 17 (858 views)
Permalink
Re: Network Traffic Collection [In reply to]

Splunk is an amazing tool and did an awesome thing and introduced a free license in 4.3.

I'm using it at two sites now and I'm loving it!

On 2012-02-23, at 3:34 PM, Mike Lyon wrote:

> Random thought, anyone ever used Splunk for this kind of thing?
>
> -mike
>
> Sent from my iPhone
>
> On Feb 23, 2012, at 10:30, Suresh Rajagopalan <sraja97 [at] gmail> wrote:
>
>> On Thu, Feb 23, 2012 at 12:19 PM, Maverick <myeaddress [at] gmail> wrote:
>>> I want to be able to see information like how much traffic an ip send
>>> over a period of time, what machines it talked to etc from this
>>> perspective it should be IP based but I would really like to know how
>>> other people do it.
>>>
>>
>>
>> Run argus on a span port.
>>
>> -Suresh
>>
>


jeroen at unfix

Feb 23, 2012, 12:52 PM

Post #8 of 17 (854 views)
Permalink
Re: Network Traffic Collection [In reply to]

On 2012-02-23 21:34 , Mike Lyon wrote:
> Random thought, anyone ever used Splunk for this kind of thing?

Various folks have, the problem of course comes down to processing
power, thus you'll need to throw a lot of hardware against it to be able
to process traffic in a decent network.

Check http://www.raffy.ch/ and http://pixlcloud.com/ etc for more
details about this.

Greets,
Jeroen


mike.lyon at gmail

Feb 23, 2012, 1:18 PM

Post #9 of 17 (856 views)
Permalink
Re: Network Traffic Collection [In reply to]

Run it with hadoop in EC2?

Sent from my iPhone

On Feb 23, 2012, at 10:52, Jeroen Massar <jeroen [at] unfix> wrote:

> On 2012-02-23 21:34 , Mike Lyon wrote:
>> Random thought, anyone ever used Splunk for this kind of thing?
>
> Various folks have, the problem of course comes down to processing
> power, thus you'll need to throw a lot of hardware against it to be able
> to process traffic in a decent network.
>
> Check http://www.raffy.ch/ and http://pixlcloud.com/ etc for more
> details about this.
>
> Greets,
> Jeroen


streiner at cluebyfour

Feb 23, 2012, 1:59 PM

Post #10 of 17 (859 views)
Permalink
Re: Network Traffic Collection [In reply to]

On Thu, 23 Feb 2012, Maverick wrote:

> I want to be able to see information like how much traffic an ip send
> over a period of time, what machines it talked to etc from this
> perspective it should be IP based but I would really like to know how
> other people do it.

Truth is that most people probably don't do it, beyond temporary, ad-hoc
deployments, to solve a specific problem at a specific point in time.
Traffic capture and analysis doesn't scale too well into multi-Gb/s
service provider environments.

Netflow tools are an option if 'reasonably accurate' is good enough for
your needs.

jms


surfer at mauigateway

Feb 23, 2012, 2:51 PM

Post #11 of 17 (851 views)
Permalink
Re: Network Traffic Collection [In reply to]

----------- myeaddress [at] gmail wrote: ----------
From: Maverick <myeaddress [at] gmail>

>> It might be an effort to write a customized traffic analysis tool like
>> wireshark with only required functionality. I would really appreciate


I want to be able to see information like how much traffic an ip send
over a period of time, what machines it talked to etc from this
perspective it should be IP based but I would really like to know how
other people do it.
-------------------------------------------------


Wouldn't Wireshark provide this for you? In particular, the "Conversations"
tool under the "Statistics" drop down menu? It adds data to the tool in
real time. If you want a graphical output the I/O graphs also under the
"Statistics" menu can graph all, or slices of the data in the main
Wireshark output.

scott


carlos at race

Feb 23, 2012, 3:30 PM

Post #12 of 17 (855 views)
Permalink
Re: Network Traffic Collection [In reply to]

Netflow / Sflow with one of the fallowing software packages

http://www.plixer.com/products/netflow-sflow/scrutinizer-netflow-sflow.php
http://www.solarwinds.com/NetFlow

http://www.arbornetworks.com/
Or the hand full of other open source options out there.



Carlos Alcantar
Race Communications / Race Team Member
101 Haskins Way, So. San Francisco, CA. 94080
Phone: +1 415 376 3314 / carlos [at] race / http://www.race.com





-----Original Message-----
From: Maverick <myeaddress [at] gmail>
Date: Thu, 23 Feb 2012 15:19:24 -0500
To: Jeroen Massar <jeroen [at] unfix>
Cc: "nanog [at] nanog" <nanog [at] nanog>
Subject: Re: Network Traffic Collection

I want to be able to see information like how much traffic an ip send
over a period of time, what machines it talked to etc from this
perspective it should be IP based but I would really like to know how
other people do it.

Best,
Ali

On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen [at] unfix> wrote:
> On 2012-02-23 21:11 , Maverick wrote:
>> Hello,
>>
>> I am trying to collect traffic traffic from pcap file and store it in
>> a database but really confused how to organize it. Should I organize
>> it on connection basis/ flow basis or IP basis.
>>
>> It might be an effort to write a customized traffic analysis tool like
>> wireshark with only required functionality. I would really appreciate
>> if someone can give me direction on write way of organizing the data
>> because right now I only see individual packets and no way of putting
>> them in some order.
>
> Does this all not completely depend on what you actually want to do with
> it? You might want to start there instead of the other way around.
>
> Greets,
> Jeroen
>
Attachments: smime.p7s (5.44 KB)


peter.phaal at gmail

Feb 23, 2012, 3:41 PM

Post #13 of 17 (849 views)
Permalink
Re: Network Traffic Collection [In reply to]

On Thu, Feb 23, 2012 at 1:59 PM, Justin M. Streiner
<streiner [at] cluebyfour> wrote:
> On Thu, 23 Feb 2012, Maverick wrote:
>
>> I want to be able to see information like how much traffic an ip send
>> over a period of time, what machines it talked to etc from this
>> perspective it should be IP based but I would really like to know how
>> other people do it.
>
>
> Truth is that most people probably don't do it, beyond temporary, ad-hoc
> deployments, to solve a specific problem at a specific point in time.
> Traffic capture and analysis doesn't scale too well into multi-Gb/s service
> provider environments.
>
> Netflow tools are an option if 'reasonably accurate' is good enough for your
> needs.
>
> jms
>

For high speed switched Ethernet environments, consider using sFlow.

You can treat sFlow as remote packet capture and use Wireshark/tcpdump
for troubleshooting network traffic:

http://blog.sflow.com/2011/11/wireshark.html

Or use sFlow reporting tools to find IP sources, protocols etc.:

http://sflow.org/products/collectors.php

Which tool to choose depends on your requirements.


owen at delong

Feb 23, 2012, 4:38 PM

Post #14 of 17 (840 views)
Permalink
Re: Network Traffic Collection [In reply to]

PCAP is not well suited to what you describe. Most people use Sflow/Cflow/...
instead.

Owen

On Feb 23, 2012, at 12:19 PM, Maverick wrote:

> I want to be able to see information like how much traffic an ip send
> over a period of time, what machines it talked to etc from this
> perspective it should be IP based but I would really like to know how
> other people do it.
>
> Best,
> Ali
>
> On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen [at] unfix> wrote:
>> On 2012-02-23 21:11 , Maverick wrote:
>>> Hello,
>>>
>>> I am trying to collect traffic traffic from pcap file and store it in
>>> a database but really confused how to organize it. Should I organize
>>> it on connection basis/ flow basis or IP basis.
>>>
>>> It might be an effort to write a customized traffic analysis tool like
>>> wireshark with only required functionality. I would really appreciate
>>> if someone can give me direction on write way of organizing the data
>>> because right now I only see individual packets and no way of putting
>>> them in some order.
>>
>> Does this all not completely depend on what you actually want to do with
>> it? You might want to start there instead of the other way around.
>>
>> Greets,
>> Jeroen
>>


mukom.tamon at gmail

Feb 24, 2012, 11:27 PM

Post #15 of 17 (823 views)
Permalink
Re: Network Traffic Collection [In reply to]

On Fri, Feb 24, 2012 at 12:20 AM, Matlock, Kenneth L
<MatlockK [at] exempla> wrote:
> Netflow + netflow collector.

+1 This guide should give you a good start.

http://techowto.files.wordpress.com/2008/09/ntop-guide.pdf

Regards

--
Mukom Akong Tamon
______________

"If we can't BREATH, we'll die. Yet, we don't LIVE in order to breath.
Ditto we SHOULDN'T WORK just to MAKE MONEY. Doing so puts us on a one
way street to IRRELEVANCE."


[In Search of Excellence & Perfection] - http://perfexcellence.org
[Moments of TechXcellence] - http://techexcellence.net
[ICT Business Integration] - http://ibiztech.wordpress.com
[About Me] - http://about.me/perfexcellence


myeaddress at gmail

Feb 25, 2012, 6:14 AM

Post #16 of 17 (824 views)
Permalink
Re: Network Traffic Collection [In reply to]

Thanks Mukom for the wonderful guide, this is really helpful. I have
few questions about ntop though.

How can I get access to the log files generated by ntop and do my own
parsing rather than looking for webbased results that are generated.
Are there any programs available that do parsing of ntops log files.
When I run ntop on pcap I don't get the throughput graphs as rrd
doesn't work on pcap is there any work around for that.

Thanks,
Ali

On Sat, Feb 25, 2012 at 2:27 AM, Mukom Akong T. <mukom.tamon [at] gmail> wrote:
> On Fri, Feb 24, 2012 at 12:20 AM, Matlock, Kenneth L
> <MatlockK [at] exempla> wrote:
>> Netflow + netflow collector.
>
> +1 This guide should give you a good start.
>
> http://techowto.files.wordpress.com/2008/09/ntop-guide.pdf
>
> Regards
>
> --
> Mukom Akong Tamon
> ______________
>
> "If we can't BREATH, we'll die. Yet, we don't LIVE in order to breath.
> Ditto we SHOULDN'T WORK just to MAKE MONEY. Doing so puts us on a one
> way street to IRRELEVANCE."
>
>
> [In Search of Excellence & Perfection] - http://perfexcellence.org
> [Moments of TechXcellence] - http://techexcellence.net
> [ICT Business Integration] -http://ibiztech.wordpress.com
> [About Me] - http://about.me/perfexcellence


mukom.tamon at gmail

Mar 2, 2012, 11:44 PM

Post #17 of 17 (763 views)
Permalink
Re: Network Traffic Collection [In reply to]

Hi Ali


On Sat, Feb 25, 2012 at 6:14 PM, Maverick <myeaddress [at] gmail> wrote:
> Thanks Mukom for the wonderful guide, this is really helpful. I have
> few questions about ntop though.
>
> How can I get access to the log files generated by ntop and do my own
> parsing rather than looking for webbased results that are generated.

It's been a while i looked under the hood of ntop. Remember that ntop
itself usually needs to be 'fed' traffic to analyse. I have never done
it myself but if I needed the raw data, I'd mirror a port and capture
it with tcpdump into a pcap file (watch disk space!!) the use whatever
analysis tool suits my needs to look at it.

> Are there any programs available that do parsing of ntops log files.
> When I run ntop on pcap I don't get the throughput graphs as rrd
> doesn't work on pcap is there any work around for that.

Not to my knowledge no. I think there's a switch (-f) for reading data
from a pcap file as opposed to a live feed. I have never played with
that as well.

There are other (possible more feature laden) commercial flow
collectors and analysers out there). I also started following trisul
earlier on in the project, you might want to check it out.



>
> Thanks,
> Ali
>
> On Sat, Feb 25, 2012 at 2:27 AM, Mukom Akong T. <mukom.tamon [at] gmail> wrote:
>> On Fri, Feb 24, 2012 at 12:20 AM, Matlock, Kenneth L
>> <MatlockK [at] exempla> wrote:
>>> Netflow + netflow collector.
>>
>> +1 This guide should give you a good start.
>>
>> http://techowto.files.wordpress.com/2008/09/ntop-guide.pdf
>>
>> Regards
>>
>> --
>> Mukom Akong Tamon
>> ______________
>>
>> "If we can't BREATH, we'll die. Yet, we don't LIVE in order to breath.
>> Ditto we SHOULDN'T WORK just to MAKE MONEY. Doing so puts us on a one
>> way street to IRRELEVANCE."
>>
>>
>> [In Search of Excellence & Perfection] - http://perfexcellence.org
>> [Moments of TechXcellence] - http://techexcellence.net
>> [ICT Business Integration] - http://ibiztech.wordpress.com
>> [About Me] - http://about.me/perfexcellence



--
Mukom Akong [Tamon]
______________

“We don't LIVE in order to BREATH. Similarly WORKING in order to make
MONEY puts us on a one way street to irrelevance.“


[In Search of Excellence & Perfection] - http://perfexcellence.org
[Moments of TechXcellence] - http://techexcellence.net
[ICT Business Integration] - http://ibiztech.wordpress.com
[About Me] - http://about.me/perfexcellence

NANOG users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.