Mailing List Archive: NTop: Users

high CPU load / can't keep up with netflow since upgrade to 3.3.8

Index | Next | Previous | View Threaded

Ggatten at waddell

May 11, 2009, 9:07 AM

Post #1 of 9 (865 views)
Permalink

high CPU load / can't keep up with netflow since upgrade to 3.3.8

Been running 3.2.1 for years and CPU has always been fine. Avg, load on
FreeBSD 6.0 is around .5, CPU around 40%, and udp port queues (netstat
-a) are most always zero, but sometimes a couple get a little backed up
for a few mins here and there.

Now running 3.3.8 compiled from ports. Load now 1.8, CPU 100%, and
most all udp netflow queues are maxed and not emptying, especially
within one instance that has many flow exporters - about 160'ish:

$ netstat -a | grep flow

Ntop Instance 1

udp4 41200 0 *.netflow-Regn29 *.*

udp4 41496 0 *.netflow-Regn16 *.*

udp4 41456 0 *.netflow-Regn15 *.*

udp4 40904 0 *.netflow-Regn11 *.*

udp4 41544 0 *.netflow-Regn10 *.*

udp4 41456 0 *.netflow-Regn02 *.*

udp4 41552 0 *.netflow-Regn88 *.*

udp4 40656 0 *.netflow-Regn53 *.*

Ntop Instance 2

udp4 0 0 *.netflow-Inet *.*

Ntop Instance 3

udp4 0 0 *.netflow-LANCore *.*

3.2.1 had its issues, but as far as netflow and cpu load it was fine.
I'd like to get on a more recent version, but keep hitting roadblocks.
Tried this back with 3.3.1 (I think?) with similar results.

Nothing in logs I find interesting. Only interesting thing on compile
what something about "sched.h":

checking sys/sched.h presence... yes

configure: WARNING: sys/sched.h: present but cannot be compiled

configure: WARNING: sys/sched.h: check for missing prerequisite
headers?

configure: WARNING: sys/sched.h: see the Autoconf documentation

configure: WARNING: sys/sched.h: section "Present But Cannot Be
Compiled"

configure: WARNING: sys/sched.h: proceeding with the preprocessor's
result

configure: WARNING: sys/sched.h: in the future, the compiler will take
precedence

I'm willing to do anything reasonable to help get this resolved. Any
help would be greatly appreciated!!!

TIA!

Gary

<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

Ggatten at waddell

May 11, 2009, 1:27 PM

Post #2 of 9 (815 views)
Permalink

Re: high CPU load / can't keep up with netflow since upgrade to 3.3.8 [In reply to]

What's diff between 3.2.x and 3.3.x netflow plugin? I deleted the
prefsCache.db and started adding netflow devices 1 at a time. Each
device as between 10 and 30 netflow exporters, and not high volume ones;
all T1 sites with ~ 15 users each.

Anyway, all is going well until I hit 3 or 4 netflow devices, then cpu
maxes out and queues start filling up. Number of flows / sec / min is
as stable as a production environment can be - not drastic changes that
would account for this high of load.

If there something different with the threads? It seems to be acting
differently - just a hunch.

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 11:08 AM
To: 'ntop [at] unipi'; ntop-dev [at] unipi
Subject: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Been running 3.2.1 for years and CPU has always been fine. Avg, load on
FreeBSD 6.0 is around .5, CPU around 40%, and udp port queues (netstat
-a) are most always zero, but sometimes a couple get a little backed up
for a few mins here and there.

Now running 3.3.8 compiled from ports. Load now 1.8, CPU 100%, and
most all udp netflow queues are maxed and not emptying, especially
within one instance that has many flow exporters - about 160'ish:

$ netstat -a | grep flow

Ntop Instance 1

udp4 41200 0 *.netflow-Regn29 *.*

udp4 41496 0 *.netflow-Regn16 *.*

udp4 41456 0 *.netflow-Regn15 *.*

udp4 40904 0 *.netflow-Regn11 *.*

udp4 41544 0 *.netflow-Regn10 *.*

udp4 41456 0 *.netflow-Regn02 *.*

udp4 41552 0 *.netflow-Regn88 *.*

udp4 40656 0 *.netflow-Regn53 *.*

Ntop Instance 2

udp4 0 0 *.netflow-Inet *.*

Ntop Instance 3

udp4 0 0 *.netflow-LANCore *.*

3.2.1 had its issues, but as far as netflow and cpu load it was fine.
I'd like to get on a more recent version, but keep hitting roadblocks.
Tried this back with 3.3.1 (I think?) with similar results.

Nothing in logs I find interesting. Only interesting thing on compile
what something about "sched.h":

checking sys/sched.h presence... yes

configure: WARNING: sys/sched.h: present but cannot be compiled

configure: WARNING: sys/sched.h: check for missing prerequisite
headers?

configure: WARNING: sys/sched.h: see the Autoconf documentation

configure: WARNING: sys/sched.h: section "Present But Cannot Be
Compiled"

configure: WARNING: sys/sched.h: proceeding with the preprocessor's
result

configure: WARNING: sys/sched.h: in the future, the compiler will take
precedence

I'm willing to do anything reasonable to help get this resolved. Any
help would be greatly appreciated!!!

TIA!

Gary

<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

Ggatten at waddell

May 11, 2009, 4:41 PM

Post #3 of 9 (817 views)
Permalink

Re: high CPU load / can't keep up with netflow since upgrade to 3.3.8 [In reply to]

Thanks for all the prompt replies on this! :-)

Looks like the additional load is related to SNMP. I commented out and
undef'd stuff related to "HAVE_SNMP" and SO FAR my CPU load is back to
"normal" and the udp queues are being serviced promptly. Not a whole
lot of traffic right now though, so will let you know for sure EOD
tomorrow - I know you're all anxiously waiting!

G

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 3:27 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

What's diff between 3.2.x and 3.3.x netflow plugin? I deleted the
prefsCache.db and started adding netflow devices 1 at a time. Each
device as between 10 and 30 netflow exporters, and not high volume ones;
all T1 sites with ~ 15 users each.

Anyway, all is going well until I hit 3 or 4 netflow devices, then cpu
maxes out and queues start filling up. Number of flows / sec / min is
as stable as a production environment can be - not drastic changes that
would account for this high of load.

If there something different with the threads? It seems to be acting
differently - just a hunch.

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 11:08 AM
To: 'ntop [at] unipi'; ntop-dev [at] unipi
Subject: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Been running 3.2.1 for years and CPU has always been fine. Avg, load on
FreeBSD 6.0 is around .5, CPU around 40%, and udp port queues (netstat
-a) are most always zero, but sometimes a couple get a little backed up
for a few mins here and there.

Now running 3.3.8 compiled from ports. Load now 1.8, CPU 100%, and
most all udp netflow queues are maxed and not emptying, especially
within one instance that has many flow exporters - about 160'ish:

$ netstat -a | grep flow

Ntop Instance 1

udp4 41200 0 *.netflow-Regn29 *.*

udp4 41496 0 *.netflow-Regn16 *.*

udp4 41456 0 *.netflow-Regn15 *.*

udp4 40904 0 *.netflow-Regn11 *.*

udp4 41544 0 *.netflow-Regn10 *.*

udp4 41456 0 *.netflow-Regn02 *.*

udp4 41552 0 *.netflow-Regn88 *.*

udp4 40656 0 *.netflow-Regn53 *.*

Ntop Instance 2

udp4 0 0 *.netflow-Inet *.*

Ntop Instance 3

udp4 0 0 *.netflow-LANCore *.*

3.2.1 had its issues, but as far as netflow and cpu load it was fine.
I'd like to get on a more recent version, but keep hitting roadblocks.
Tried this back with 3.3.1 (I think?) with similar results.

Nothing in logs I find interesting. Only interesting thing on compile
what something about "sched.h":

checking sys/sched.h presence... yes

configure: WARNING: sys/sched.h: present but cannot be compiled

configure: WARNING: sys/sched.h: check for missing prerequisite
headers?

configure: WARNING: sys/sched.h: see the Autoconf documentation

configure: WARNING: sys/sched.h: section "Present But Cannot Be
Compiled"

configure: WARNING: sys/sched.h: proceeding with the preprocessor's
result

configure: WARNING: sys/sched.h: in the future, the compiler will take
precedence

I'm willing to do anything reasonable to help get this resolved. Any
help would be greatly appreciated!!!

TIA!

Gary

<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

deri at ntop

May 12, 2009, 6:57 AM

Post #4 of 9 (808 views)
Permalink

Re: high CPU load / can't keep up with netflow since upgrade to 3.3.8 [In reply to]

Gary
did you start ntop -i none ? (i.e. is't because of NetFlow or any
other pcap-related issues?)

Luca

On May 11, 2009, at 6:07 PM, Gary Gatten wrote:

> Been running 3.2.1 for years and CPU has always been fine. Avg,
> load on FreeBSD 6.0 is around .5, CPU around 40%, and udp port
> queues (netstat �a) are most always zero, but sometimes a couple get
> a little backed up for a few mins here and there.
>
> Now running 3.3.8 compiled from ports. Load now 1.8, CPU 100%, and
> most all udp netflow queues are maxed and not emptying, especially
> within one instance that has many flow exporters � about 160�ish:
>
> $ netstat -a | grep flow
> Ntop Instance 1
> udp4 41200 0 *.netflow-Regn29 *.*
> udp4 41496 0 *.netflow-Regn16 *.*
> udp4 41456 0 *.netflow-Regn15 *.*
> udp4 40904 0 *.netflow-Regn11 *.*
> udp4 41544 0 *.netflow-Regn10 *.*
> udp4 41456 0 *.netflow-Regn02 *.*
> udp4 41552 0 *.netflow-Regn88 *.*
> udp4 40656 0 *.netflow-Regn53 *.*
>
> Ntop Instance 2
> udp4 0 0 *.netflow-Inet *.*
>
> Ntop Instance 3
> udp4 0 0 *.netflow-LANCore *.*
>
> 3.2.1 had its issues, but as far as netflow and cpu load it was
> fine. I�d like to get on a more recent version, but keep hitting
> roadblocks. Tried this back with 3.3.1 (I think?) with similar
> results.
>
> Nothing in logs I find interesting. Only interesting thing on
> compile what something about �sched.h�:
>
> checking sys/sched.h presence... yes
> configure: WARNING: sys/sched.h: present but cannot be compiled
> configure: WARNING: sys/sched.h: check for missing prerequisite
> headers?
> configure: WARNING: sys/sched.h: see the Autoconf documentation
> configure: WARNING: sys/sched.h: section "Present But Cannot Be
> Compiled"
> configure: WARNING: sys/sched.h: proceeding with the preprocessor's
> result
> configure: WARNING: sys/sched.h: in the future, the compiler will
> take precedence
>
>
> I�m willing to do anything reasonable to help get this resolved.
> Any help would be greatly appreciated!!!
>
> TIA!
>
> Gary
>
> "This email is intended to be reviewed by only the intended
> recipient and may contain information that is privileged and/or
> confidential. If you are not the intended recipient, you are hereby
> notified that any review, use, dissemination, disclosure or copying
> of this email and its attachments, if any, is strictly prohibited.
> If you have received this email in error, please immediately notify
> the sender by return email and delete this email from your system."
> _______________________________________________
> Ntop mailing list
> Ntop [at] unipi
> http://listgateway.unipi.it/mailman/listinfo/ntop

---
If you can not measure it, you can not improve it - Lord Kelvin

Ggatten at waddell

May 12, 2009, 1:56 PM

Post #5 of 9 (809 views)
Permalink

Re: high CPU load / can't keep up with netflow since upgrade to 3.3.8 [In reply to]

OK, under production load I'm having similar issues even with
"HAVE_SNMP" undef'd. I tweaked some timers in globals-defines.h that is
causing higher than typical memory usage. I don't have much RAM (768MB)
so I'm doing a lot of swapping. This is part of the problem for sure,
so at this point I'm not 100% sure if the SNMP (UtilLoop) had anything
to do with this and if so how much? I do recall when first trying to
get on 3.3.x some time ago I didn't tweak the timers as much, but I also
had even less RAM then! More testing, but prolly not today!

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 6:42 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Thanks for all the prompt replies on this! :-)

Looks like the additional load is related to SNMP. I commented out and
undef'd stuff related to "HAVE_SNMP" and SO FAR my CPU load is back to
"normal" and the udp queues are being serviced promptly. Not a whole
lot of traffic right now though, so will let you know for sure EOD
tomorrow - I know you're all anxiously waiting!

G

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 3:27 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

What's diff between 3.2.x and 3.3.x netflow plugin? I deleted the
prefsCache.db and started adding netflow devices 1 at a time. Each
device as between 10 and 30 netflow exporters, and not high volume ones;
all T1 sites with ~ 15 users each.

Anyway, all is going well until I hit 3 or 4 netflow devices, then cpu
maxes out and queues start filling up. Number of flows / sec / min is
as stable as a production environment can be - not drastic changes that
would account for this high of load.

If there something different with the threads? It seems to be acting
differently - just a hunch.

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 11:08 AM
To: 'ntop [at] unipi'; ntop-dev [at] unipi
Subject: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Been running 3.2.1 for years and CPU has always been fine. Avg, load on
FreeBSD 6.0 is around .5, CPU around 40%, and udp port queues (netstat
-a) are most always zero, but sometimes a couple get a little backed up
for a few mins here and there.

Now running 3.3.8 compiled from ports. Load now 1.8, CPU 100%, and
most all udp netflow queues are maxed and not emptying, especially
within one instance that has many flow exporters - about 160'ish:

$ netstat -a | grep flow

Ntop Instance 1

udp4 41200 0 *.netflow-Regn29 *.*

udp4 41496 0 *.netflow-Regn16 *.*

udp4 41456 0 *.netflow-Regn15 *.*

udp4 40904 0 *.netflow-Regn11 *.*

udp4 41544 0 *.netflow-Regn10 *.*

udp4 41456 0 *.netflow-Regn02 *.*

udp4 41552 0 *.netflow-Regn88 *.*

udp4 40656 0 *.netflow-Regn53 *.*

Ntop Instance 2

udp4 0 0 *.netflow-Inet *.*

Ntop Instance 3

udp4 0 0 *.netflow-LANCore *.*

3.2.1 had its issues, but as far as netflow and cpu load it was fine.
I'd like to get on a more recent version, but keep hitting roadblocks.
Tried this back with 3.3.1 (I think?) with similar results.

Nothing in logs I find interesting. Only interesting thing on compile
what something about "sched.h":

checking sys/sched.h presence... yes

configure: WARNING: sys/sched.h: present but cannot be compiled

configure: WARNING: sys/sched.h: check for missing prerequisite
headers?

configure: WARNING: sys/sched.h: see the Autoconf documentation

configure: WARNING: sys/sched.h: section "Present But Cannot Be
Compiled"

configure: WARNING: sys/sched.h: proceeding with the preprocessor's
result

configure: WARNING: sys/sched.h: in the future, the compiler will take
precedence

I'm willing to do anything reasonable to help get this resolved. Any
help would be greatly appreciated!!!

TIA!

Gary

<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

Ggatten at waddell

May 12, 2009, 1:57 PM

Post #6 of 9 (812 views)
Permalink

Re: high CPU load / can't keep up with netflow since upgrade to 3.3.8 [In reply to]

PS: On 3.3.9 from SourceForge now - got tired of the "[warn] kevent: Bad
file descriptor" errors.

G

________________________________

From: Gary Gatten
Sent: Tuesday, May 12, 2009 3:56 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

OK, under production load I'm having similar issues even with
"HAVE_SNMP" undef'd. I tweaked some timers in globals-defines.h that is
causing higher than typical memory usage. I don't have much RAM (768MB)
so I'm doing a lot of swapping. This is part of the problem for sure,
so at this point I'm not 100% sure if the SNMP (UtilLoop) had anything
to do with this and if so how much? I do recall when first trying to
get on 3.3.x some time ago I didn't tweak the timers as much, but I also
had even less RAM then! More testing, but prolly not today!

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 6:42 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Thanks for all the prompt replies on this! :-)

Looks like the additional load is related to SNMP. I commented out and
undef'd stuff related to "HAVE_SNMP" and SO FAR my CPU load is back to
"normal" and the udp queues are being serviced promptly. Not a whole
lot of traffic right now though, so will let you know for sure EOD
tomorrow - I know you're all anxiously waiting!

G

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 3:27 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

What's diff between 3.2.x and 3.3.x netflow plugin? I deleted the
prefsCache.db and started adding netflow devices 1 at a time. Each
device as between 10 and 30 netflow exporters, and not high volume ones;
all T1 sites with ~ 15 users each.

Anyway, all is going well until I hit 3 or 4 netflow devices, then cpu
maxes out and queues start filling up. Number of flows / sec / min is
as stable as a production environment can be - not drastic changes that
would account for this high of load.

If there something different with the threads? It seems to be acting
differently - just a hunch.

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 11:08 AM
To: 'ntop [at] unipi'; ntop-dev [at] unipi
Subject: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Been running 3.2.1 for years and CPU has always been fine. Avg, load on
FreeBSD 6.0 is around .5, CPU around 40%, and udp port queues (netstat
-a) are most always zero, but sometimes a couple get a little backed up
for a few mins here and there.

Now running 3.3.8 compiled from ports. Load now 1.8, CPU 100%, and
most all udp netflow queues are maxed and not emptying, especially
within one instance that has many flow exporters - about 160'ish:

$ netstat -a | grep flow

Ntop Instance 1

udp4 41200 0 *.netflow-Regn29 *.*

udp4 41496 0 *.netflow-Regn16 *.*

udp4 41456 0 *.netflow-Regn15 *.*

udp4 40904 0 *.netflow-Regn11 *.*

udp4 41544 0 *.netflow-Regn10 *.*

udp4 41456 0 *.netflow-Regn02 *.*

udp4 41552 0 *.netflow-Regn88 *.*

udp4 40656 0 *.netflow-Regn53 *.*

Ntop Instance 2

udp4 0 0 *.netflow-Inet *.*

Ntop Instance 3

udp4 0 0 *.netflow-LANCore *.*

3.2.1 had its issues, but as far as netflow and cpu load it was fine.
I'd like to get on a more recent version, but keep hitting roadblocks.
Tried this back with 3.3.1 (I think?) with similar results.

Nothing in logs I find interesting. Only interesting thing on compile
what something about "sched.h":

checking sys/sched.h presence... yes

configure: WARNING: sys/sched.h: present but cannot be compiled

configure: WARNING: sys/sched.h: check for missing prerequisite
headers?

configure: WARNING: sys/sched.h: see the Autoconf documentation

configure: WARNING: sys/sched.h: section "Present But Cannot Be
Compiled"

configure: WARNING: sys/sched.h: proceeding with the preprocessor's
result

configure: WARNING: sys/sched.h: in the future, the compiler will take
precedence

I'm willing to do anything reasonable to help get this resolved. Any
help would be greatly appreciated!!!

TIA!

Gary

<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

Ggatten at waddell

May 13, 2009, 9:28 AM

Post #7 of 9 (806 views)
Permalink

Re: high CPU load / can't keep up with netflow since upgrade to 3.3.8 [In reply to]

I've had one instance running for about an hour now and my netflow udp
queues are still maxed out. This instance has about 160 flow exporters,
but rate of flows (flows/sec) is not massive or anything. I'd guess
it's about the same as another instance I have for the HQ LAN, which
it's queue is empty! The diff is the LAN instance is not tracking
remote hosts....

So, could the additional overhead / load be related to name resolution
or more specifically GeoIP? I don't think GeoIP was around in 3.2, so
maybe that whole process is slowing things down enough to cause me
problems, especially on initial startup when there are a bunch of new
hosts to discover info about?

I'm hoping once it catches up everything will be fine?

________________________________

From: Gary Gatten
Sent: Tuesday, May 12, 2009 3:57 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

PS: On 3.3.9 from SourceForge now - got tired of the "[warn] kevent: Bad
file descriptor" errors.

G

________________________________

From: Gary Gatten
Sent: Tuesday, May 12, 2009 3:56 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

OK, under production load I'm having similar issues even with
"HAVE_SNMP" undef'd. I tweaked some timers in globals-defines.h that is
causing higher than typical memory usage. I don't have much RAM (768MB)
so I'm doing a lot of swapping. This is part of the problem for sure,
so at this point I'm not 100% sure if the SNMP (UtilLoop) had anything
to do with this and if so how much? I do recall when first trying to
get on 3.3.x some time ago I didn't tweak the timers as much, but I also
had even less RAM then! More testing, but prolly not today!

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 6:42 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Thanks for all the prompt replies on this! :-)

Looks like the additional load is related to SNMP. I commented out and
undef'd stuff related to "HAVE_SNMP" and SO FAR my CPU load is back to
"normal" and the udp queues are being serviced promptly. Not a whole
lot of traffic right now though, so will let you know for sure EOD
tomorrow - I know you're all anxiously waiting!

G

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 3:27 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

What's diff between 3.2.x and 3.3.x netflow plugin? I deleted the
prefsCache.db and started adding netflow devices 1 at a time. Each
device as between 10 and 30 netflow exporters, and not high volume ones;
all T1 sites with ~ 15 users each.

Anyway, all is going well until I hit 3 or 4 netflow devices, then cpu
maxes out and queues start filling up. Number of flows / sec / min is
as stable as a production environment can be - not drastic changes that
would account for this high of load.

If there something different with the threads? It seems to be acting
differently - just a hunch.

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 11:08 AM
To: 'ntop [at] unipi'; ntop-dev [at] unipi
Subject: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Been running 3.2.1 for years and CPU has always been fine. Avg, load on
FreeBSD 6.0 is around .5, CPU around 40%, and udp port queues (netstat
-a) are most always zero, but sometimes a couple get a little backed up
for a few mins here and there.

Now running 3.3.8 compiled from ports. Load now 1.8, CPU 100%, and
most all udp netflow queues are maxed and not emptying, especially
within one instance that has many flow exporters - about 160'ish:

$ netstat -a | grep flow

Ntop Instance 1

udp4 41200 0 *.netflow-Regn29 *.*

udp4 41496 0 *.netflow-Regn16 *.*

udp4 41456 0 *.netflow-Regn15 *.*

udp4 40904 0 *.netflow-Regn11 *.*

udp4 41544 0 *.netflow-Regn10 *.*

udp4 41456 0 *.netflow-Regn02 *.*

udp4 41552 0 *.netflow-Regn88 *.*

udp4 40656 0 *.netflow-Regn53 *.*

Ntop Instance 2

udp4 0 0 *.netflow-Inet *.*

Ntop Instance 3

udp4 0 0 *.netflow-LANCore *.*

3.2.1 had its issues, but as far as netflow and cpu load it was fine.
I'd like to get on a more recent version, but keep hitting roadblocks.
Tried this back with 3.3.1 (I think?) with similar results.

Nothing in logs I find interesting. Only interesting thing on compile
what something about "sched.h":

checking sys/sched.h presence... yes

configure: WARNING: sys/sched.h: present but cannot be compiled

configure: WARNING: sys/sched.h: check for missing prerequisite
headers?

configure: WARNING: sys/sched.h: see the Autoconf documentation

configure: WARNING: sys/sched.h: section "Present But Cannot Be
Compiled"

configure: WARNING: sys/sched.h: proceeding with the preprocessor's
result

configure: WARNING: sys/sched.h: in the future, the compiler will take
precedence

I'm willing to do anything reasonable to help get this resolved. Any
help would be greatly appreciated!!!

TIA!

Gary

<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

Ggatten at waddell

May 13, 2009, 1:44 PM

Post #8 of 9 (807 views)
Permalink

Re: high CPU load / can't keep up with netflow since upgrade to 3.3.8 [In reply to]

OK, after more hours of testing....

I undid a bunch of my tweaks to globals-defines.h and recompiled
everything, same problem. RAM usage was far less as expected, but cpu
still maxed and netflow queues always full. I recompiled my 3.2.1 stuff
with the same tweaks and it runs perfect, queues empty almost
immediately after startup and my cpu is averaging about 15% right now,
load avg is .58!

I did notice this instance is using 20 threads on 3.2.1 whereas it was
only.... 13 or so on 3.3.9.

On 3.3.9 I've: undef'd HAVE_SNMP, disabled rrd, and tried to disable
GeoIP by renaming the .dat files. Any help would be appreciated - spent
at least 40 hours on this so far!

Thanks!

Gary

________________________________

From: Gary Gatten
Sent: Tuesday, May 12, 2009 3:57 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

PS: On 3.3.9 from SourceForge now - got tired of the "[warn] kevent: Bad
file descriptor" errors.

G

________________________________

From: Gary Gatten
Sent: Tuesday, May 12, 2009 3:56 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

OK, under production load I'm having similar issues even with
"HAVE_SNMP" undef'd. I tweaked some timers in globals-defines.h that is
causing higher than typical memory usage. I don't have much RAM (768MB)
so I'm doing a lot of swapping. This is part of the problem for sure,
so at this point I'm not 100% sure if the SNMP (UtilLoop) had anything
to do with this and if so how much? I do recall when first trying to
get on 3.3.x some time ago I didn't tweak the timers as much, but I also
had even less RAM then! More testing, but prolly not today!

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 6:42 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Thanks for all the prompt replies on this! :-)

Looks like the additional load is related to SNMP. I commented out and
undef'd stuff related to "HAVE_SNMP" and SO FAR my CPU load is back to
"normal" and the udp queues are being serviced promptly. Not a whole
lot of traffic right now though, so will let you know for sure EOD
tomorrow - I know you're all anxiously waiting!

G

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 3:27 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

What's diff between 3.2.x and 3.3.x netflow plugin? I deleted the
prefsCache.db and started adding netflow devices 1 at a time. Each
device as between 10 and 30 netflow exporters, and not high volume ones;
all T1 sites with ~ 15 users each.

Anyway, all is going well until I hit 3 or 4 netflow devices, then cpu
maxes out and queues start filling up. Number of flows / sec / min is
as stable as a production environment can be - not drastic changes that
would account for this high of load.

If there something different with the threads? It seems to be acting
differently - just a hunch.

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 11:08 AM
To: 'ntop [at] unipi'; ntop-dev [at] unipi
Subject: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Been running 3.2.1 for years and CPU has always been fine. Avg, load on
FreeBSD 6.0 is around .5, CPU around 40%, and udp port queues (netstat
-a) are most always zero, but sometimes a couple get a little backed up
for a few mins here and there.

Now running 3.3.8 compiled from ports. Load now 1.8, CPU 100%, and
most all udp netflow queues are maxed and not emptying, especially
within one instance that has many flow exporters - about 160'ish:

$ netstat -a | grep flow

Ntop Instance 1

udp4 41200 0 *.netflow-Regn29 *.*

udp4 41496 0 *.netflow-Regn16 *.*

udp4 41456 0 *.netflow-Regn15 *.*

udp4 40904 0 *.netflow-Regn11 *.*

udp4 41544 0 *.netflow-Regn10 *.*

udp4 41456 0 *.netflow-Regn02 *.*

udp4 41552 0 *.netflow-Regn88 *.*

udp4 40656 0 *.netflow-Regn53 *.*

Ntop Instance 2

udp4 0 0 *.netflow-Inet *.*

Ntop Instance 3

udp4 0 0 *.netflow-LANCore *.*

3.2.1 had its issues, but as far as netflow and cpu load it was fine.
I'd like to get on a more recent version, but keep hitting roadblocks.
Tried this back with 3.3.1 (I think?) with similar results.

Nothing in logs I find interesting. Only interesting thing on compile
what something about "sched.h":

checking sys/sched.h presence... yes

configure: WARNING: sys/sched.h: present but cannot be compiled

configure: WARNING: sys/sched.h: check for missing prerequisite
headers?

configure: WARNING: sys/sched.h: see the Autoconf documentation

configure: WARNING: sys/sched.h: section "Present But Cannot Be
Compiled"

configure: WARNING: sys/sched.h: proceeding with the preprocessor's
result

configure: WARNING: sys/sched.h: in the future, the compiler will take
precedence

I'm willing to do anything reasonable to help get this resolved. Any
help would be greatly appreciated!!!

TIA!

Gary

<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

Ggatten at waddell

May 15, 2009, 10:45 AM

Post #9 of 9 (789 views)
Permalink

Re: high CPU load / can't keep up with netflow since upgrade to 3.3.8 [In reply to]

Giving up on this - either something really stupid or way over my head -
or maybe both! 3.2.1 does what I NEED for the most part, so I guess I'm
stuck with it until I get some better hardware. Not sure where all the
extra overhead came from with 3.3, but my load averages from top are
between 4 ant 10 times greater than with 3.2 under roughly the same
netflow load, with same args and defs in globals-defines.h. When I get
better hardware hopefully it will "fix" my problems!

Gary

________________________________

From: Gary Gatten
Sent: Wednesday, May 13, 2009 3:45 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

OK, after more hours of testing....

I undid a bunch of my tweaks to globals-defines.h and recompiled
everything, same problem. RAM usage was far less as expected, but cpu
still maxed and netflow queues always full. I recompiled my 3.2.1 stuff
with the same tweaks and it runs perfect, queues empty almost
immediately after startup and my cpu is averaging about 15% right now,
load avg is .58!

I did notice this instance is using 20 threads on 3.2.1 whereas it was
only.... 13 or so on 3.3.9.

On 3.3.9 I've: undef'd HAVE_SNMP, disabled rrd, and tried to disable
GeoIP by renaming the .dat files. Any help would be appreciated - spent
at least 40 hours on this so far!

Thanks!

Gary

________________________________

From: Gary Gatten
Sent: Tuesday, May 12, 2009 3:57 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

PS: On 3.3.9 from SourceForge now - got tired of the "[warn] kevent: Bad
file descriptor" errors.

G

________________________________

From: Gary Gatten
Sent: Tuesday, May 12, 2009 3:56 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

OK, under production load I'm having similar issues even with
"HAVE_SNMP" undef'd. I tweaked some timers in globals-defines.h that is
causing higher than typical memory usage. I don't have much RAM (768MB)
so I'm doing a lot of swapping. This is part of the problem for sure,
so at this point I'm not 100% sure if the SNMP (UtilLoop) had anything
to do with this and if so how much? I do recall when first trying to
get on 3.3.x some time ago I didn't tweak the timers as much, but I also
had even less RAM then! More testing, but prolly not today!

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 6:42 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Thanks for all the prompt replies on this! :-)

Looks like the additional load is related to SNMP. I commented out and
undef'd stuff related to "HAVE_SNMP" and SO FAR my CPU load is back to
"normal" and the udp queues are being serviced promptly. Not a whole
lot of traffic right now though, so will let you know for sure EOD
tomorrow - I know you're all anxiously waiting!

G

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 3:27 PM
To: 'ntop [at] unipi'; 'ntop-dev [at] unipi'
Subject: RE: high CPU load / can't keep up with netflow since upgrade to
3.3.8

What's diff between 3.2.x and 3.3.x netflow plugin? I deleted the
prefsCache.db and started adding netflow devices 1 at a time. Each
device as between 10 and 30 netflow exporters, and not high volume ones;
all T1 sites with ~ 15 users each.

Anyway, all is going well until I hit 3 or 4 netflow devices, then cpu
maxes out and queues start filling up. Number of flows / sec / min is
as stable as a production environment can be - not drastic changes that
would account for this high of load.

If there something different with the threads? It seems to be acting
differently - just a hunch.

________________________________

From: Gary Gatten
Sent: Monday, May 11, 2009 11:08 AM
To: 'ntop [at] unipi'; ntop-dev [at] unipi
Subject: high CPU load / can't keep up with netflow since upgrade to
3.3.8

Been running 3.2.1 for years and CPU has always been fine. Avg, load on
FreeBSD 6.0 is around .5, CPU around 40%, and udp port queues (netstat
-a) are most always zero, but sometimes a couple get a little backed up
for a few mins here and there.

Now running 3.3.8 compiled from ports. Load now 1.8, CPU 100%, and
most all udp netflow queues are maxed and not emptying, especially
within one instance that has many flow exporters - about 160'ish:

$ netstat -a | grep flow

Ntop Instance 1

udp4 41200 0 *.netflow-Regn29 *.*

udp4 41496 0 *.netflow-Regn16 *.*

udp4 41456 0 *.netflow-Regn15 *.*

udp4 40904 0 *.netflow-Regn11 *.*

udp4 41544 0 *.netflow-Regn10 *.*

udp4 41456 0 *.netflow-Regn02 *.*

udp4 41552 0 *.netflow-Regn88 *.*

udp4 40656 0 *.netflow-Regn53 *.*

Ntop Instance 2

udp4 0 0 *.netflow-Inet *.*

Ntop Instance 3

udp4 0 0 *.netflow-LANCore *.*

3.2.1 had its issues, but as far as netflow and cpu load it was fine.
I'd like to get on a more recent version, but keep hitting roadblocks.
Tried this back with 3.3.1 (I think?) with similar results.

Nothing in logs I find interesting. Only interesting thing on compile
what something about "sched.h":

checking sys/sched.h presence... yes

configure: WARNING: sys/sched.h: present but cannot be compiled

configure: WARNING: sys/sched.h: check for missing prerequisite
headers?

configure: WARNING: sys/sched.h: see the Autoconf documentation

configure: WARNING: sys/sched.h: section "Present But Cannot Be
Compiled"

configure: WARNING: sys/sched.h: proceeding with the preprocessor's
result

configure: WARNING: sys/sched.h: in the future, the compiler will take
precedence

I'm willing to do anything reasonable to help get this resolved. Any
help would be greatly appreciated!!!

TIA!

Gary

<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads