Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NTop: Misc

Re: strange traffic level drop over time (Justin Azoff)

 

 

NTop misc RSS feed   Index | Next | Previous | View Threaded


jovimon at gmail

Mar 3, 2012, 10:43 AM

Post #1 of 6 (604 views)
Permalink
Re: strange traffic level drop over time (Justin Azoff)

Hello,
Few months ago we were testing PF_RING and found a similar behaviour.

We were using 6x snort and noticed that in few hours (or maybe a day, don't
remember exactly) the traffic level dropped to only around 100Mbps, when
the switch graphs showed more than 700 Mbps on the wire.

Back then, we thougt it was a problem with our NIC (bnx2 driver)
and discarded PF_RING usage. If you find a solution to this problem, I
think we will be able to retest again to see if it works better.

Thank you very much,

Jose Vila.


On Fri, Mar 2, 2012 at 12:00, <ntop-misc-request [at] listgateway>wrote:

> Message: 1
> Date: Thu, 1 Mar 2012 09:24:49 -0500
> From: Justin Azoff <JAzoff [at] albany>
> To: ntop-misc [at] listgateway
> Subject: Re: [Ntop-misc] strange traffic level drop over time
> Message-ID: <20120301142449.GW15524 [at] datacomm>
> Content-Type: text/plain; charset=us-ascii
>
> On Thu, Mar 01, 2012 at 09:32:50AM +0100, Luca Deri wrote:
> > Justin
> > Via the pfring DAQ we inject drop rules in the kernel whenever snort
> > detects bad traffic. It might be that these rules are causing the
> > problem you see as all the rules are removed when you restart snort.
> > Can you please check what traffic is dropped and verify if my feelings
> > are correct?
>
> I'm not sure how to figure out what traffic is actually being dropped,
> but I did notice this:
>
> tucker:/proc/net/pf_ring# grep 'Rules' 2*;echo;sleep 5;grep 'Rules' 2*
> 20568-sniff1.63:# Sw Filt. Rules : 24261
> 20568-sniff1.63:# Hw Filt. Rules : 0
> 20571-sniff1.64:# Sw Filt. Rules : 24390
> 20571-sniff1.64:# Hw Filt. Rules : 0
>
> 20568-sniff1.63:# Sw Filt. Rules : 24415
> 20568-sniff1.63:# Hw Filt. Rules : 0
> 20571-sniff1.64:# Sw Filt. Rules : 24528
> 20571-sniff1.64:# Hw Filt. Rules : 0
>
> is that what you are referring to?
>
> I'm not sure why the daq module should do this in the first place..
> especially when not running inline.
>
> --
> -- Justin Azoff
> -- Network Security & Performance Analyst
>
>
> ------------------------------
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
> End of Ntop-misc Digest, Vol 93, Issue 2
> ****************************************
>


JAzoff at albany

Mar 3, 2012, 11:08 AM

Post #2 of 6 (597 views)
Permalink
Re: strange traffic level drop over time (Justin Azoff) [In reply to]

On Sat, Mar 03, 2012 at 07:43:00PM +0100, Jose Vila wrote:
> Back then, we thougt it was a problem with our NIC (bnx2 driver) and
> discarded PF_RING usage. If you find a solution to this problem, I
> think we will be able to retest again to see if it works better.

In daq_pfring.c there is a block of code in a switch statement leading
up to:

rc = pfring_handle_hash_filtering_rule(context->ring_handle, &hash_rule, 1 /* add_rule */);

I removed this whole section and rebuilt the daq module. That made the
problem go away. I'm not 100% sure why the daq module does this in the
first place. Bug or feature, removing it makes everything work great.

--
-- Justin Azoff
-- Network Security & Performance Analyst
_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc


jnebrera at eneotecnologia

Mar 3, 2012, 12:14 PM

Post #3 of 6 (600 views)
Permalink
Re: strange traffic level drop over time (Justin Azoff) [In reply to]

Hi Justin,

I believe the idea is to reduce the load on snort when you know it going to drop a packet

That is, snort receives a stream until decides it's a drop. Further packets in that flow will be dropped directly at pfring layer (or even at hardware layer if you are lucky and own a card that supports hardware filters)

This of course makes snort see less packets, and thus allows snort more room for extra traffic

This means, without it you might be able to process a stream of 500Mbps were you might drop 100. With it enable you might process a stream of up to 600 Mbps and drop those 100 at a lower layer (so snort processes the same amount of traffic)

Your call, coherent info or extra performance

If you don't have drop rules, this won't help you. If your traffic is very clean, neither but when things are nasty is very valuable

Btw, you can get those stats at the interface level and thus have the same info

Enviado desde mi iPhone

El 03/03/2012, a las 20:08, Justin Azoff <JAzoff [at] albany> escribió:

> On Sat, Mar 03, 2012 at 07:43:00PM +0100, Jose Vila wrote:
>> Back then, we thougt it was a problem with our NIC (bnx2 driver) and
>> discarded PF_RING usage. If you find a solution to this problem, I
>> think we will be able to retest again to see if it works better.
>
> In daq_pfring.c there is a block of code in a switch statement leading
> up to:
>
> rc = pfring_handle_hash_filtering_rule(context->ring_handle, &hash_rule, 1 /* add_rule */);
>
> I removed this whole section and rebuilt the daq module. That made the
> problem go away. I'm not 100% sure why the daq module does this in the
> first place. Bug or feature, removing it makes everything work great.
>
> --
> -- Justin Azoff
> -- Network Security & Performance Analyst
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc


JAzoff at albany

Mar 3, 2012, 1:17 PM

Post #4 of 6 (602 views)
Permalink
Re: strange traffic level drop over time (Justin Azoff) [In reply to]

On Sat, Mar 03, 2012 at 09:14:51PM +0100, Jaime Nebrera wrote:
> Hi Justin,
>
> I believe the idea is to reduce the load on snort when you know it going to drop a packet
> That is, snort receives a stream until decides it's a drop. Further packets in that flow will be dropped directly at pfring layer (or even at hardware layer if you are lucky and own a card that supports hardware filters)
>
> This of course makes snort see less packets, and thus allows snort more room for extra traffic

The problems being that

* Snort is not running inline so it isn't actually dropping packets
* Other things are running on the box that need to see all the packets
* The feature does not appear to even work anyway, over time the
majority of the traffic ends up being dropped.

4/5ths of our traffic is definitely not alert traffic, so I'm not really
sure what was being dropped. Is there a way to print out the current
pf_ring filter?

so.. bug of feature, I think this needs to be an option that you can
turn off. Even if it worked 100% correctly, it would still break the
other apps running on the box.

--
-- Justin Azoff
-- Network Security & Performance Analyst
_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc


jnebrera at eneotecnologia

Mar 3, 2012, 11:00 PM

Post #5 of 6 (587 views)
Permalink
Re: strange traffic level drop over time (Justin Azoff) [In reply to]

Hi Justin,

I agree with you this should be configurable but is still valid in IDS mode

Your problem is not the fact you are running as IDS but that other apps require to see that traffic

Enviado desde mi iPhone

El 03/03/2012, a las 22:17, Justin Azoff <JAzoff [at] albany> escribió:

> On Sat, Mar 03, 2012 at 09:14:51PM +0100, Jaime Nebrera wrote:
>> Hi Justin,
>>
>> I believe the idea is to reduce the load on snort when you know it going to drop a packet
>> That is, snort receives a stream until decides it's a drop. Further packets in that flow will be dropped directly at pfring layer (or even at hardware layer if you are lucky and own a card that supports hardware filters)
>>
>> This of course makes snort see less packets, and thus allows snort more room for extra traffic
>
> The problems being that
>
> * Snort is not running inline so it isn't actually dropping packets
> * Other things are running on the box that need to see all the packets
> * The feature does not appear to even work anyway, over time the
> majority of the traffic ends up being dropped.
>
> 4/5ths of our traffic is definitely not alert traffic, so I'm not really
> sure what was being dropped. Is there a way to print out the current
> pf_ring filter?
>
> so.. bug of feature, I think this needs to be an option that you can
> turn off. Even if it worked 100% correctly, it would still break the
> other apps running on the box.
>
> --
> -- Justin Azoff
> -- Network Security & Performance Analyst
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc


deri at ntop

Mar 5, 2012, 5:59 AM

Post #6 of 6 (596 views)
Permalink
Re: strange traffic level drop over time (Justin Azoff) [In reply to]

Hi all
we have made a patch to PF_RING DAQ. Please update and let us know if it
works now. Note that we have also updated the README file under the same
directory to explain all the options.

Thanks Luca

On 03/04/2012 08:00 AM, Jaime Nebrera wrote:
> Hi Justin,
>
> I agree with you this should be configurable but is still valid in IDS mode
>
> Your problem is not the fact you are running as IDS but that other apps require to see that traffic
>
> Enviado desde mi iPhone
>
> El 03/03/2012, a las 22:17, Justin Azoff<JAzoff [at] albany> escribió:
>
>> On Sat, Mar 03, 2012 at 09:14:51PM +0100, Jaime Nebrera wrote:
>>> Hi Justin,
>>>
>>> I believe the idea is to reduce the load on snort when you know it going to drop a packet
>>> That is, snort receives a stream until decides it's a drop. Further packets in that flow will be dropped directly at pfring layer (or even at hardware layer if you are lucky and own a card that supports hardware filters)
>>>
>>> This of course makes snort see less packets, and thus allows snort more room for extra traffic
>> The problems being that
>>
>> * Snort is not running inline so it isn't actually dropping packets
>> * Other things are running on the box that need to see all the packets
>> * The feature does not appear to even work anyway, over time the
>> majority of the traffic ends up being dropped.
>>
>> 4/5ths of our traffic is definitely not alert traffic, so I'm not really
>> sure what was being dropped. Is there a way to print out the current
>> pf_ring filter?
>>
>> so.. bug of feature, I think this needs to be an option that you can
>> turn off. Even if it worked 100% correctly, it would still break the
>> other apps running on the box.
>>
>> --
>> -- Justin Azoff
>> -- Network Security& Performance Analyst
>> _______________________________________________
>> Ntop-misc mailing list
>> Ntop-misc [at] listgateway
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

NTop misc RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.