Mailing List Archive: NTop: Misc

Re: [Snort-users] Pfring crashes the kernel with white lists.

Index | Next | Previous | View Threaded

peter.bates at ucl

Jul 18, 2012, 1:36 PM

Post #1 of 13 (983 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello all - and apologies for cross-posting.

On 21/06/2012 00:58, livio Ricciulli wrote:
> It looks like the ssl dynamic processor of the latest snort
> distributions causes the DAQ verdict to be WHITE_LIST for certain
> ssl connections. This is perfectly ok if you are NOT using --daq
> pfring. If you use --daq pfring with snort 2.9.2.x, it will cause
> pfring to add a monotonically increasing number of WHITE_LIST
> pfring filters in kernel memory causing memory exhaustion and
> eventually a crash after a few hours/days/months depending on your
> traffic rate. We have a pfring distribution that fixes this and
> other problems (like supporting bpf filtering) at
> http://www.metaflows.com/pfring/PF_RING.tgz
>
> The WHITE_LIST fix is very simple; basically, if the verdict from
> the snort processing is WHITE_LIST, you set it to PASS instead in
> daq_pfring.c.
>
> We will send this fixes to the Ntop folks as well..

This bug hit me today with PF_RING from svn and Snort 2.9.2.3
- - available RAM was exhausted over the course of a couple of hours and
left me with a dead IDS (well, until I reboot it tomorrow).

I'd appreciate if the Metaflows changes could make it into the current
version of PF_RING and PF_RING DAQ - I presume there's no change in
Snort 2.9.3 that will alter this behaviour.

- --
Peter Bates
Senior Computer Security Officer Phone: +44(0)2076792049
Information Services Division Internal Ext: 32049
University College London
London WC1E 6BT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQBx41AAoJELhVoVpEMS6R16wH/ic43tGW9TIQngMdLBxezlfL
WIMhMPTrLI6CYzuacBdZ0VEHGppdyzNIg7tbubgbH2cHF6Ad69aZEKzE/g6pXLEh
4PFds/8oH7SwgWoglHcORm/xzU1PY0UKN+n80wQq9du8jtptPVCxTyg3ph0r4ZrE
YCYShzYJHPY3nUkii+rNM9nrM/+MfDNaIASaJIqCbUuLU3sNcf7JjE0Tfrof/NLU
+g5GaafaBHsKCWkcf+aivBLQ4MJt3gAJJdSseeQhYvdy8Sm6xMuuv4Rcw3yWwaPc
HYvOWd4BndXP0Pje9USsNeZa2yiZtXjmpaItWHKI/rQ4+gQF21rznJ4yp5ygbV0=
=ZIBf
-----END PGP SIGNATURE-----

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

cardigliano at ntop

Jul 19, 2012, 2:17 AM

Post #2 of 13 (961 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

Hi Peter
first of all let me clarify that it is not true that "it will cause pfring to add a monotonically increasing number of WHITE_LIST pfring filters".
In fact by default kernel rules idle for more than 5 minutes (it is possible to tune this with --daq-var kernel-filters-idle-timeout=<seconds>)
are automatically removed. Of course there can be an high number of rules if there is an high number of concurrent active flows.
Anyway we changed this in current svn, so the problem in no longer present. Please update and let us know.

Best Regards
Alfredo

On Jul 18, 2012, at 10:36 PM, Peter Bates wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> Hello all - and apologies for cross-posting.
>
> On 21/06/2012 00:58, livio Ricciulli wrote:
>> It looks like the ssl dynamic processor of the latest snort
>> distributions causes the DAQ verdict to be WHITE_LIST for certain
>> ssl connections. This is perfectly ok if you are NOT using --daq
>> pfring. If you use --daq pfring with snort 2.9.2.x, it will cause
>> pfring to add a monotonically increasing number of WHITE_LIST
>> pfring filters in kernel memory causing memory exhaustion and
>> eventually a crash after a few hours/days/months depending on your
>> traffic rate. We have a pfring distribution that fixes this and
>> other problems (like supporting bpf filtering) at
>> http://www.metaflows.com/pfring/PF_RING.tgz
>>
>> The WHITE_LIST fix is very simple; basically, if the verdict from
>> the snort processing is WHITE_LIST, you set it to PASS instead in
>> daq_pfring.c.
>>
>> We will send this fixes to the Ntop folks as well..
>
> This bug hit me today with PF_RING from svn and Snort 2.9.2.3
> - - available RAM was exhausted over the course of a couple of hours and
> left me with a dead IDS (well, until I reboot it tomorrow).
>
> I'd appreciate if the Metaflows changes could make it into the current
> version of PF_RING and PF_RING DAQ - I presume there's no change in
> Snort 2.9.3 that will alter this behaviour.
>
> - --
> Peter Bates
> Senior Computer Security Officer Phone: +44(0)2076792049
> Information Services Division Internal Ext: 32049
> University College London
> London WC1E 6BT
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJQBx41AAoJELhVoVpEMS6R16wH/ic43tGW9TIQngMdLBxezlfL
> WIMhMPTrLI6CYzuacBdZ0VEHGppdyzNIg7tbubgbH2cHF6Ad69aZEKzE/g6pXLEh
> 4PFds/8oH7SwgWoglHcORm/xzU1PY0UKN+n80wQq9du8jtptPVCxTyg3ph0r4ZrE
> YCYShzYJHPY3nUkii+rNM9nrM/+MfDNaIASaJIqCbUuLU3sNcf7JjE0Tfrof/NLU
> +g5GaafaBHsKCWkcf+aivBLQ4MJt3gAJJdSseeQhYvdy8Sm6xMuuv4Rcw3yWwaPc
> HYvOWd4BndXP0Pje9USsNeZa2yiZtXjmpaItWHKI/rQ4+gQF21rznJ4yp5ygbV0=
> =ZIBf
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

peter.bates at ucl

Jul 19, 2012, 8:32 AM

Post #3 of 13 (958 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello all

On 19/07/2012 10:17, Alfredo Cardigliano wrote:
> first of all let me clarify that it is not true that "it will cause
> pfring to add a monotonically increasing number of WHITE_LIST
> pfring filters". In fact by default kernel rules idle for more than
> 5 minutes (it is possible to tune this with --daq-var
> kernel-filters-idle-timeout=<seconds>) are automatically removed.
> Of course there can be an high number of rules if there is an high
> number of concurrent active flows. Anyway we changed this in
> current svn, so the problem in no longer present. Please update and
> let us know.

I've updated to the latest PF_RING svn, and I'm seeing a slower but
what still appears to be a problem with RAM slowly disappearing.

PF_RING Version : 5.4.5 ($Revision: 5583$)
Ring slots : 4096
Slot version : 14
Capture TX : No [RX only]
IP Defragment : No
Socket Mode : Standard
Transparent mode : Yes (mode 1)
Total rings : 8
Total plugins : 0

I've also got the ixgbe driver (non-DNA) loaded - compiled from the
latest svn.

I ran set_irq_affinity.sh on eth6 (the ixgbe interface)

The 8 rings are 8 copies of Snort (2.9.2.3):

config daq_mode: passive
config daq: pfring
config daq_var: clusterid=10
config daq_var: watermark=64
config daq_var: timeout=1

And then
- --daq-var bindcpu=i
where i=0 to 7 for my 8 cores

I can see from the /proc/net/pf_ring/*eth6* stats that
there are no filter rules.

RAM free

2708988k 15:59
1060708k 16:29

Does anyone have any suggestions as to a likely cause?

- --
Peter Bates
Senior Computer Security Officer Phone: +44(0)2076792049
Information Services Division Internal Ext: 32049
University College London
London WC1E 6BT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQCCiSAAoJELhVoVpEMS6RXrUIAKIrTdFJfuWdpoTc3Dce2ZM+
D1KEfQD46dPBYFhZwXcpMvPQKKIK7vXbIdFM8IxxKLLrC0yI9zpi6k+rwqthnWKG
PeGKCOtPPjfr/6U6tcvw6KXatbMe1C3yLf9+XX/9v5NM3Tcm4ERTXKXI06nLeX01
U66dGp+Uqf4cR6VWVvu+s5zrGcThbioUNxmlMzheVIy8FkBDEao8rR6qODOafy5q
5tH08HpjQdYxGGjuwEuBty0pEsdjaY1nkZsntEIVFZjLKjOU3AA5ORyv7I37BMn+
MCAsViTcQwxoYBXY03ITNYm089K3WJVRnhP4UJmfYdUr0ItLXCwsHEw8oRXIBio=
=YKYj
-----END PGP SIGNATURE-----

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

peter.bates at ucl

Jul 20, 2012, 3:04 AM

Post #4 of 13 (948 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear all

I just wanted to add to this and not appear
that I was being critical of PF_RING.

I've switched back to:

PF_RING Version : 5.4.3 ($Revision: 5443$)

and I'm now running Snort x 8 with transparent_mode=1 (ixgbe, non-DNA),
a BPF filter and each instance bound to a CPU (--daq-var bindcpu=x)
without RAM trickling away and a subsequent kernel crash.

Whatever my problem is it is present in the 5.4.4 stable release
and the latest svn version.

- --
Peter Bates
Senior Computer Security Officer Phone: +44(0)2076792049
Information Services Division Internal Ext: 32049
University College London
London WC1E 6BT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQCS0gAAoJELhVoVpEMS6RwlsIAJOLrMNIumpgkiRCDd1ShEu6
+rSNmI5G8HjzndzGSJ7a6BhIv7ck7WopipW0anrYkxR2uy/R3VS1mpRl9ow3I2fX
J69yKKbebV2zNVm+mSMawoQJ+NsQm6omkODnpy+LcRsyZYzkLjj8mH4dxNzEJ0GJ
2JV62b9Z/KMPflfa6bal7ryz1gPsj0Xv+bUB1DkwcLKtGHcIuQxA0R5wZ3TehvAX
EZc20vhSQR6FdObJfP+kCmhALa1qyXfMtsitPLu/gOhwJoD8eIDAukNLVSiZVUbo
tFGjjSwtr8rNL1fCYwI0wgT6uDt5fiai84k2J9zfm3SXaOKKkZ7n1KxzHJP4Qg4=
=NroB
-----END PGP SIGNATURE-----

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

deri at ntop

Jul 20, 2012, 3:07 AM

Post #5 of 13 (950 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

Peter,
did you try Alfredo's patch that was disabling filtering rule injection for PASS verdicts?

Luca

On Jul 20, 2012, at 12:04 PM, Peter Bates wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> Dear all
>
> I just wanted to add to this and not appear
> that I was being critical of PF_RING.
>
> I've switched back to:
>
> PF_RING Version : 5.4.3 ($Revision: 5443$)
>
> and I'm now running Snort x 8 with transparent_mode=1 (ixgbe, non-DNA),
> a BPF filter and each instance bound to a CPU (--daq-var bindcpu=x)
> without RAM trickling away and a subsequent kernel crash.
>
> Whatever my problem is it is present in the 5.4.4 stable release
> and the latest svn version.
>
> - --
> Peter Bates
> Senior Computer Security Officer Phone: +44(0)2076792049
> Information Services Division Internal Ext: 32049
> University College London
> London WC1E 6BT
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJQCS0gAAoJELhVoVpEMS6RwlsIAJOLrMNIumpgkiRCDd1ShEu6
> +rSNmI5G8HjzndzGSJ7a6BhIv7ck7WopipW0anrYkxR2uy/R3VS1mpRl9ow3I2fX
> J69yKKbebV2zNVm+mSMawoQJ+NsQm6omkODnpy+LcRsyZYzkLjj8mH4dxNzEJ0GJ
> 2JV62b9Z/KMPflfa6bal7ryz1gPsj0Xv+bUB1DkwcLKtGHcIuQxA0R5wZ3TehvAX
> EZc20vhSQR6FdObJfP+kCmhALa1qyXfMtsitPLu/gOhwJoD8eIDAukNLVSiZVUbo
> tFGjjSwtr8rNL1fCYwI0wgT6uDt5fiai84k2J9zfm3SXaOKKkZ7n1KxzHJP4Qg4=
> =NroB
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

peter.bates at ucl

Jul 20, 2012, 3:14 AM

Post #6 of 13 (949 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello all

On 20/07/2012 11:07, Luca Deri wrote:
> Peter, did you try Alfredo's patch that was disabling filtering
> rule injection for PASS verdicts?

Yes - same problem - 2.6.x kernel.

/proc/net/pf_ring/*ethx* showed that the Sw. Filters
were disabled after I updated but the slow depletion of RAM continued.

I could be wrong but it even looked to me like RAM was disappearing
slowly just by insmod'ing the later PF_RING and PF_RING-aware ixgbe
- - not even running an application - but that might have been my
imagination.

- --
Peter Bates
Senior Computer Security Officer Phone: +44(0)2076792049
Information Services Division Internal Ext: 32049
University College London
London WC1E 6BT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQCS+AAAoJELhVoVpEMS6RgqsH/iBQ3HEv+60zHJ44SFTGMUNn
XbJ+kv78GPbQdrDAngsrSwrXv6SSI9YxGHn3skJm+acv69zla5tEOmbGmLk0plJi
ijJpcSt9Gs/2D+fZEMFDBfx464GnjsaVNC6mAz0WplhIOKM7z4CcKoE0O+RqzZOl
IQRDzIAQj0w8HnWBABm3jB4bQ4mMOGRHCjj91uTMiN26lca4mooitpEBmDw1gdXU
iyfsN4xe+PDGCJUIPFPwwepTFvKhO2OaLRgfJJuN2k7A/BA0eedOdLSroAFq74HM
DplLyBi+hY4i/LVbPVlwwhXa6Riklxzj0caP8w1lGNyQtHEGXAQK5G9/cwdBcmc=
=gsH+
-----END PGP SIGNATURE-----

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

deri at ntop

Jul 20, 2012, 3:19 AM

Post #7 of 13 (948 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

Peter
is the ram decreasing in the kernel or in user-space?

Luca

On Jul 20, 2012, at 12:14 PM, Peter Bates wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> Hello all
>
> On 20/07/2012 11:07, Luca Deri wrote:
>> Peter, did you try Alfredo's patch that was disabling filtering
>> rule injection for PASS verdicts?
>
> Yes - same problem - 2.6.x kernel.
>
> /proc/net/pf_ring/*ethx* showed that the Sw. Filters
> were disabled after I updated but the slow depletion of RAM continued.
>
> I could be wrong but it even looked to me like RAM was disappearing
> slowly just by insmod'ing the later PF_RING and PF_RING-aware ixgbe
> - - not even running an application - but that might have been my
> imagination.
>
> - --
> Peter Bates
> Senior Computer Security Officer Phone: +44(0)2076792049
> Information Services Division Internal Ext: 32049
> University College London
> London WC1E 6BT
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJQCS+AAAoJELhVoVpEMS6RgqsH/iBQ3HEv+60zHJ44SFTGMUNn
> XbJ+kv78GPbQdrDAngsrSwrXv6SSI9YxGHn3skJm+acv69zla5tEOmbGmLk0plJi
> ijJpcSt9Gs/2D+fZEMFDBfx464GnjsaVNC6mAz0WplhIOKM7z4CcKoE0O+RqzZOl
> IQRDzIAQj0w8HnWBABm3jB4bQ4mMOGRHCjj91uTMiN26lca4mooitpEBmDw1gdXU
> iyfsN4xe+PDGCJUIPFPwwepTFvKhO2OaLRgfJJuN2k7A/BA0eedOdLSroAFq74HM
> DplLyBi+hY4i/LVbPVlwwhXa6Riklxzj0caP8w1lGNyQtHEGXAQK5G9/cwdBcmc=
> =gsH+
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

peter.bates at ucl

Jul 20, 2012, 3:24 AM

Post #8 of 13 (949 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 20/07/2012 11:19, Luca Deri wrote:
> Peter is the ram decreasing in the kernel or in user-space?

A question a bit above my head, I think!

I mean the 'free' count shown in the output of 'free' or 'top'.

It drained away and then the server required power cycling -
the oops on the screen referred to pf_ring and ixgbe
but I didn't catch all of the details.

On the other occasions I stopped everything
when I saw the RAM ticking down and rmmod'ed PF_RING and ixgbe.

- --
Peter Bates
Senior Computer Security Officer Phone: +44(0)2076792049
Information Services Division Internal Ext: 32049
University College London
London WC1E 6BT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQCTHlAAoJELhVoVpEMS6Rh2sH/jAxhQa6ea6vUYFY4y0f1mkO
KUaYIYfdAfakrQYRnoTbtpBRHbnCkOBlNtGO3oiK/gzqzWMj61gxmIVYhNCzhWoK
nWgI0BzuaambYlMBbNVALd9683xvRu84yFVm6ppD4W8t0H3/A/V8oIlCOVD0D1h5
P3GAkhFY5aURn3Ol8GPlFHdVBKiCiNaqkOuJ3v/UVq2Zv1kyM5CHl6cThDsdlvqw
PnB9MggrKYpnFeTm+IFLMfRI09LxCvojXVYuOa87tm4RUofFXys5RYc2ByBrYp/5
oUw/1uYHnxyWi9TsbPGUO1mETQHU2c1mh6ELD7jwV5dxXpmtlXB40Gw1SI4xpw0=
=E1yp
-----END PGP SIGNATURE-----

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

deri at ntop

Jul 20, 2012, 3:30 AM

Post #9 of 13 (953 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

I just want to know if the supposed leak is on the kernel (PF_RING) or user-space (snort). If is in the kernel, and you have no rules, then we have a leak = bug. If snort leaks memory then I cannot do much

Luca

On Jul 20, 2012, at 12:24 PM, Peter Bates wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> On 20/07/2012 11:19, Luca Deri wrote:
>> Peter is the ram decreasing in the kernel or in user-space?
>
> A question a bit above my head, I think!
>
> I mean the 'free' count shown in the output of 'free' or 'top'.
>
> It drained away and then the server required power cycling -
> the oops on the screen referred to pf_ring and ixgbe
> but I didn't catch all of the details.
>
> On the other occasions I stopped everything
> when I saw the RAM ticking down and rmmod'ed PF_RING and ixgbe.
>
> - --
> Peter Bates
> Senior Computer Security Officer Phone: +44(0)2076792049
> Information Services Division Internal Ext: 32049
> University College London
> London WC1E 6BT
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJQCTHlAAoJELhVoVpEMS6Rh2sH/jAxhQa6ea6vUYFY4y0f1mkO
> KUaYIYfdAfakrQYRnoTbtpBRHbnCkOBlNtGO3oiK/gzqzWMj61gxmIVYhNCzhWoK
> nWgI0BzuaambYlMBbNVALd9683xvRu84yFVm6ppD4W8t0H3/A/V8oIlCOVD0D1h5
> P3GAkhFY5aURn3Ol8GPlFHdVBKiCiNaqkOuJ3v/UVq2Zv1kyM5CHl6cThDsdlvqw
> PnB9MggrKYpnFeTm+IFLMfRI09LxCvojXVYuOa87tm4RUofFXys5RYc2ByBrYp/5
> oUw/1uYHnxyWi9TsbPGUO1mETQHU2c1mh6ELD7jwV5dxXpmtlXB40Gw1SI4xpw0=
> =E1yp
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

peter.bates at ucl

Jul 20, 2012, 3:56 AM

Post #10 of 13 (965 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 20/07/2012 11:30, Luca Deri wrote:
> I just want to know if the supposed leak is on the kernel (PF_RING)
> or user-space (snort). If is in the kernel, and you have no rules,
> then we have a leak = bug. If snort leaks memory then I cannot do
> much

Okay, I will try some testing and report back.

If there was a memory leak in Snort I still don't understand why
changing back to a previous version of PF_RING seems to remove the
problem.

All I've been doing is:

Rebuild PF_RING kernel driver + make install
Rebuild PF_RING-aware ixgbe + install
Rebuild userland/snort/pfring-daq-module + install

insmod pf_ring and ixgbe
and then starting Snort.

- --
Peter Bates
Senior Computer Security Officer Phone: +44(0)2076792049
Information Services Division Internal Ext: 32049
University College London
London WC1E 6BT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQCTlfAAoJELhVoVpEMS6R9egH/1bz4CAYVM303s1LuIdH5sfY
hzXxJfyGG0JaFkH3xAnN/wn8IEx5qS/DuVfwSFgKgaNyt3jsQSPGRU4R8spiid9W
DJUbdbtnhMmG8suwuYF3Xa64WNUknZmr1nCZvVtAzNawyxYaQGp7an1KpQPwVL34
ZxvkgXFlrMZlDYBDQU0SeQtmpS6d1YtSVM8Mr+105S24bFw9DHjs11bOgbc4u3RY
9UdqIVoqq6QqsQ8dKXNsFoWPC9ByeeNN/DPm7eSkydEO2fXmvYggRC7n+bP7q7LM
Ocn6dRY7k9NIAPo9U+jHhsg8M4HBuoMnl+x0HjUKY85YzIcURgiVgc6OXp/Zytk=
=9jOD
-----END PGP SIGNATURE-----

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

peter.bates at ucl

Jul 20, 2012, 4:25 AM

Post #11 of 13 (951 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello again

Okay, I've compiled the latest PF_RING from svn.

I realised I had 'httpry' linked against PF_RING, so I've tried that.

PF_RING Version : 5.4.5 ($Revision: 5588$)

Free RAM:

2565544 11:48
2565896 11:56
2556920 12:04

No application running, just PF_RING and ixgbe, fine.

Start httpry - there is a complaint that the kernel is using slot
version 14 but the application has been built for slot version 13
so presumably at this point it falls back to standard NAPI capture?

2272220 12:07
1948304 12:11
1588240 12:15
1086440 12:20

In this case /proc/net/pf_ring/x-eth6.5 is created but the packet
stats are empty.

I don't see errors about different slot versions when I run Snort but
the entries in /proc/net/pf_ring do indicate software filters and
packet counts/losses.

- --
Peter Bates
Senior Computer Security Officer Phone: +44(0)2076792049
Information Services Division Internal Ext: 32049
University College London
London WC1E 6BT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQCUAZAAoJELhVoVpEMS6RxFUIAKTVejDroXB4etmXrWJoiacz
c28TUgcdjiCInEcRkJfkTh/XbYbqXWXYQ3iSDCPkDMMk8UCTv/LzPMMjM95HXh6H
iGCEwu4ekyTzsA3KJL/ZTx7gPbRQYXnFQEms5uEHZskbyVtM+GZ7DrHvnnLulIYL
ShAeK2PQMKGUBwEWwG10NMsnLJYfYKdZHjFSDQiIpmbZuguaTShV/5YxRbslnY8i
2n+pkO8Ghz6B2TvDIeMmL5C9VUSXzqxJbuAHWTVXYBXb3a63Q3RxfhQfFuThdg8B
BadESOhE/6T68LkTbQwS3xbFlfbNsN4noOLSl5+kP/lH4xEY0MhGrRD+TCIqmyY=
=hXBi
-----END PGP SIGNATURE-----

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

cardigliano at ntop

Jul 20, 2012, 9:48 AM

Post #12 of 13 (947 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

Hello Peter
I'm trying to reproduce your issues but without success until now, any additional information about you configuration would be helpful (e.g. commands, traffic info, etc.).
Thank you.

Now for instance I'm running 4 snort instances IDS on top of PF_RING-aware ixgbe drivers with transparent_mode=1
Free RAM
3621944 18:05
3621952 18:42

Beside "free" see also "cat /proc/slabinfo" for kernel memory info.

Best Regards
Alfredo

On Jul 20, 2012, at 1:25 PM, Peter Bates wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> Hello again
>
> Okay, I've compiled the latest PF_RING from svn.
>
> I realised I had 'httpry' linked against PF_RING, so I've tried that.
>
> PF_RING Version : 5.4.5 ($Revision: 5588$)
>
> Free RAM:
>
> 2565544 11:48
> 2565896 11:56
> 2556920 12:04
>
> No application running, just PF_RING and ixgbe, fine.
>
> Start httpry - there is a complaint that the kernel is using slot
> version 14 but the application has been built for slot version 13
> so presumably at this point it falls back to standard NAPI capture?
>
> 2272220 12:07
> 1948304 12:11
> 1588240 12:15
> 1086440 12:20
>
> In this case /proc/net/pf_ring/x-eth6.5 is created but the packet
> stats are empty.
>
> I don't see errors about different slot versions when I run Snort but
> the entries in /proc/net/pf_ring do indicate software filters and
> packet counts/losses.
>
> - --
> Peter Bates
> Senior Computer Security Officer Phone: +44(0)2076792049
> Information Services Division Internal Ext: 32049
> University College London
> London WC1E 6BT
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJQCUAZAAoJELhVoVpEMS6RxFUIAKTVejDroXB4etmXrWJoiacz
> c28TUgcdjiCInEcRkJfkTh/XbYbqXWXYQ3iSDCPkDMMk8UCTv/LzPMMjM95HXh6H
> iGCEwu4ekyTzsA3KJL/ZTx7gPbRQYXnFQEms5uEHZskbyVtM+GZ7DrHvnnLulIYL
> ShAeK2PQMKGUBwEWwG10NMsnLJYfYKdZHjFSDQiIpmbZuguaTShV/5YxRbslnY8i
> 2n+pkO8Ghz6B2TvDIeMmL5C9VUSXzqxJbuAHWTVXYBXb3a63Q3RxfhQfFuThdg8B
> BadESOhE/6T68LkTbQwS3xbFlfbNsN4noOLSl5+kP/lH4xEY0MhGrRD+TCIqmyY=
> =hXBi
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

peter.bates at ucl

Jul 23, 2012, 6:25 AM

Post #13 of 13 (929 views)
Permalink

Re: [Snort-users] Pfring crashes the kernel with white lists. [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello all

On 20/07/2012 17:48, Alfredo Cardigliano wrote:
> I'm trying to reproduce your issues but without success until now,
> any additional information about you configuration would be helpful
> (e.g. commands, traffic info, etc.).

I've tried some more testing, and it does just look like my system
behaves a bit odd but does actually keep running - so maybe I was
premature. It did definitely crash in the kernel last week though.

Using PF_RING fresh from svn today:

My routine:

modprobe ixgbe InterruptThrottleRate=4000
modprobe pf_ring transparent_mode=1 min_num_slots=4096 enable_tx_capture=0
ifconfig eth6 up
ifconfig eth6 promisc
set_irq_affinity.sh eth6
no rx vectors found on eth6
no tx vectors found on eth6
eth6 mask=1 for /proc/irq/61/smp_affinity
eth6 mask=2 for /proc/irq/62/smp_affinity
eth6 mask=4 for /proc/irq/63/smp_affinity
eth6 mask=8 for /proc/irq/64/smp_affinity
eth6 mask=10 for /proc/irq/65/smp_affinity
eth6 mask=20 for /proc/irq/66/smp_affinity
eth6 mask=40 for /proc/irq/67/smp_affinity
eth6 mask=80 for /proc/irq/68/smp_affinity

Run Snort x 8, clusterid=10 --daq-var bindcpu=(0 to 7)
Also running 2 other applications not compiled with PF_RING or that
fail to create anything in /proc/net/pf_ring as there's not other
RAM/slots.

pfcount:
=========================
Absolute Stats: [4376008 pkts rcvd][208825 pkts dropped]
Total Pkts=4584833/Dropped=4.6 %
4'376'008 pkts - 3'235'083'544 bytes [273'018.38 pkt/sec - 1'614.69
Mbit/sec]
=========================
Actual Stats: 260806 pkts [1'000.11 ms][260'777.31 pps/1.60 Gbps]
=========================

Free RAM

3594740 12:46 (before adding ixgbe, pf_ring)
2679276 13:00
2023816 13:10
1489004 13:21
1124252 13:30
570344 13:40
162148 13:50
123344 13:55
122140 14:00
120104 14:10
129132 14:15
121124 14:20

top - 14:00:51 up 1:25, 3 users, load average: 8.09, 6.87, 6.14
Tasks: 223 total, 10 running, 213 sleeping, 0 stopped, 0 zombie
Cpu0 : 11.3%us, 0.6%sy, 0.0%ni, 68.1%id, 0.7%wa, 0.0%hi, 19.2%si,
0.0%st
Cpu1 : 13.9%us, 0.7%sy, 0.0%ni, 59.3%id, 0.3%wa, 0.0%hi, 25.8%si,
0.0%st
Cpu2 : 9.8%us, 0.5%sy, 0.0%ni, 58.3%id, 0.6%wa, 0.0%hi, 30.8%si,
0.0%st
Cpu3 : 11.7%us, 0.7%sy, 0.0%ni, 67.9%id, 0.3%wa, 0.0%hi, 19.4%si,
0.0%st
Cpu4 : 12.4%us, 0.6%sy, 0.0%ni, 64.1%id, 0.7%wa, 0.0%hi, 22.2%si,
0.0%st
Cpu5 : 11.6%us, 0.7%sy, 0.0%ni, 67.0%id, 0.3%wa, 0.0%hi, 20.3%si,
0.0%st
Cpu6 : 10.9%us, 0.6%sy, 0.0%ni, 60.5%id, 0.4%wa, 0.0%hi, 27.6%si,
0.0%st
Cpu7 : 9.8%us, 0.6%sy, 0.0%ni, 49.8%id, 0.2%wa, 0.0%hi, 39.6%si,
0.0%st
Mem: 4117672k total, 3993416k used, 124256k free, 27144k buffers
Swap: 2031612k total, 0k used, 2031612k free, 1842316k cached

I have the output of /proc/slabinfo at various times but I'm not sure
what figures I should look at.

- --
Peter Bates
Senior Computer Security Officer Phone: +44(0)2076792049
Information Services Division Internal Ext: 32049
University College London
London WC1E 6BT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQDVDcAAoJELhVoVpEMS6RqRQH/ja0iKzQABg8rE9XyAIBhY2M
4U7E/FqLBpUCMAwwjb8AU/jP7hfwodiMyX39/oJqUtn9cRYIAFQI+BQ3IqdcHXMj
nbYzPBCy/DLUmnSLyVsM0T+dXuf8D+eeqA9hIgUUufC1Ks4MUgZ836Cl2XqRfdsQ
f9zjIh4SCB/u+jRXWH8321+UVmr8k6K+rS/kk/CLiJT6jYThIZSK7jMAdW+M8Wqe
5vIkLQppdaoJ4kwqqeLx4tjEmI+k0F/2+YtRjJpsNMXBJeowII6svrbyhh7olg5L
XAXA9J5r14wZKdtFBIBerFyb3jgPqnrA1kdDjBSxffVdJJFS3tF/8ybpMcaApQY=
=7NKY
-----END PGP SIGNATURE-----

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads