Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NTop: Users

Track Local Hosts Abnormal CPU Usage

 

 

NTop users RSS feed   Index | Next | Previous | View Threaded


nicholas.turner at utoronto

Jun 17, 2010, 10:31 AM

Post #1 of 6 (1087 views)
Permalink
Track Local Hosts Abnormal CPU Usage

Hi,

I've been implementing NTOP over the past few weeks, and overall it is
just an awesome program =) However recently I have been having some
problems with CPU Usage, which is therein causing a huge amount of
dropped packets from libpcap.

Just some background information, I am running on CentOS 5.5, and my
command line args are as follows:

-u ntop
-w 3000
-i bond0
-m 128.100.179.0/24,142.151.0.0/16,10.10.0.0/16
-L
-P /ntop/rrd
-d
-x 4000


I've been having some weird issues with NTOP shooting up to 100% CPU
usage, then libpcap dropping packets (I assume due to not having any
CPU cycles to capture them!).

My compiled installation was working fine, until I noticed that NTOP
was hitting the 8192 host maximum without the "-x" flag set. At this
point I did not notice any libpcap dropped packets. THEN, I started
playing around with the -x flag, setting it to 20000 in order to see
if I would reach that cap (I did, because I still had remote hosts
being added).

After trying -x 20000, the libpcap started to drop tons of packets, so
I removed the -x flag, going back to the 8192 default, but now for
some reason I was getting dropped packets even at this host count. I
tried cleaning out some preferences (fingerprint.db macPrefix.db
prefscache.db) and the RRD directory, but this didnt help. Since I
couldnt get NTOP back to it's original state of 8192 hosts working
fine, I played with the -x setting until I found <1% dropped packets
over an hour period at the 4000-5000 range.
(I feel I should mention that when I say "drop tons of packets" I mean
like 50%-300% although I dont quite understand how that is possible).

Now, upon perusing the man ntop page some more I find the
--track-local-hosts option, which would be perfect for our
implementation. However, upon enabling that, even though ntop is now
only seeing ~150-250 local hosts, it shoots up to 100% cpu usage, and
libpcap starts to drop packets.

So my question is, is there any way to get my ntop to run
--track-local-hosts without so much CPU usage? Or are there some
inherent heavy cpu operations that are doing this? The server is a
~2.2GHz Intel, with 1GB of ram, just a backup server, but I thought it
would be able to handle ntop fine. The average network load with the
4000 host cap is ~5000 packets per second. Is this something NTOP is
able to handle with some tweaks? Or must I install PF_RING in order to
handle these packets correctly?


Thanks,

Nick


_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


Ggatten at waddell

Jun 17, 2010, 10:35 AM

Post #2 of 6 (1074 views)
Permalink
Re: Track Local Hosts Abnormal CPU Usage [In reply to]

For fun try disabling name/ip resolution.

----- Original Message -----
From: ntop-bounces [at] listgateway <ntop-bounces [at] listgateway>
To: ntop [at] listgateway <ntop [at] listgateway>
Sent: Thu Jun 17 12:31:06 2010
Subject: [Ntop] Track Local Hosts Abnormal CPU Usage

Hi,

I've been implementing NTOP over the past few weeks, and overall it is
just an awesome program =) However recently I have been having some
problems with CPU Usage, which is therein causing a huge amount of
dropped packets from libpcap.

Just some background information, I am running on CentOS 5.5, and my
command line args are as follows:

-u ntop
-w 3000
-i bond0
-m 128.100.179.0/24,142.151.0.0/16,10.10.0.0/16
-L
-P /ntop/rrd
-d
-x 4000


I've been having some weird issues with NTOP shooting up to 100% CPU
usage, then libpcap dropping packets (I assume due to not having any
CPU cycles to capture them!).

My compiled installation was working fine, until I noticed that NTOP
was hitting the 8192 host maximum without the "-x" flag set. At this
point I did not notice any libpcap dropped packets. THEN, I started
playing around with the -x flag, setting it to 20000 in order to see
if I would reach that cap (I did, because I still had remote hosts
being added).

After trying -x 20000, the libpcap started to drop tons of packets, so
I removed the -x flag, going back to the 8192 default, but now for
some reason I was getting dropped packets even at this host count. I
tried cleaning out some preferences (fingerprint.db macPrefix.db
prefscache.db) and the RRD directory, but this didnt help. Since I
couldnt get NTOP back to it's original state of 8192 hosts working
fine, I played with the -x setting until I found <1% dropped packets
over an hour period at the 4000-5000 range.
(I feel I should mention that when I say "drop tons of packets" I mean
like 50%-300% although I dont quite understand how that is possible).

Now, upon perusing the man ntop page some more I find the
--track-local-hosts option, which would be perfect for our
implementation. However, upon enabling that, even though ntop is now
only seeing ~150-250 local hosts, it shoots up to 100% cpu usage, and
libpcap starts to drop packets.

So my question is, is there any way to get my ntop to run
--track-local-hosts without so much CPU usage? Or are there some
inherent heavy cpu operations that are doing this? The server is a
~2.2GHz Intel, with 1GB of ram, just a backup server, but I thought it
would be able to handle ntop fine. The average network load with the
4000 host cap is ~5000 packets per second. Is this something NTOP is
able to handle with some tweaks? Or must I install PF_RING in order to
handle these packets correctly?


Thanks,

Nick


_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


nicholas.turner at utoronto

Jun 17, 2010, 11:13 AM

Post #3 of 6 (1047 views)
Permalink
Re: Track Local Hosts Abnormal CPU Usage [In reply to]

I assume that would be the -n flag?

Seems to have no effect on my operation with -x 4000 hosts, still same
CPU usage hovering around 60-70%. Tried -n and --track-local-hosts,
after 30 minutes of runtime, ntop has processed 1,798,000 packets, and
libpcap has dropped just shy of 10,000,000 packets, and thinks it has
only seen 197MB of traffic over this time (definitely been more, since
555% of packets have been dropped).

So I would have to assume that the name resolution of IP's is not
causing the problem, but thanks for the suggestion!

On that note, I have also tried the -b flag to disable protocol
decoding, but that did not seem to help my CPU usage.

Nick

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


Ggatten at waddell

Jun 17, 2010, 11:44 AM

Post #4 of 6 (1057 views)
Permalink
Re: Track Local Hosts Abnormal CPU Usage [In reply to]

What do you think your pps is? Can you confirm it with stats from your switch?

Run top only on the ntop PID and enable thread view. There's another arg to top I can't recall right now, but it will help show you which thread is using the CPU. I'll see if I can find / remember this.


-----Original Message-----
From: ntop-bounces [at] listgateway [mailto:ntop-bounces [at] listgateway] On Behalf Of Nicholas Turner
Sent: Thursday, June 17, 2010 1:13 PM
To: ntop [at] listgateway
Subject: Re: [Ntop] Track Local Hosts Abnormal CPU Usage

I assume that would be the -n flag?

Seems to have no effect on my operation with -x 4000 hosts, still same
CPU usage hovering around 60-70%. Tried -n and --track-local-hosts,
after 30 minutes of runtime, ntop has processed 1,798,000 packets, and
libpcap has dropped just shy of 10,000,000 packets, and thinks it has
only seen 197MB of traffic over this time (definitely been more, since
555% of packets have been dropped).

So I would have to assume that the name resolution of IP's is not
causing the problem, but thanks for the suggestion!

On that note, I have also tried the -b flag to disable protocol
decoding, but that did not seem to help my CPU usage.

Nick

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


nicholas.turner at utoronto

Jun 17, 2010, 1:12 PM

Post #5 of 6 (1052 views)
Permalink
Re: Track Local Hosts Abnormal CPU Usage [In reply to]

Both the switch and NTOP are reporting in the 5500pkt/s to 6500pkt/s,
so I think the fiber tap is working fine, NTOP seems to be seeing all
of the packets.

I tried running top -H (thread view) -p10440 (my ntop PID), but it
just shows me a bunch of lt-ntop threads... none showing too much
discernible difference that might help me (so maybe I havent got the
right arguments!) Infact the only difference seems to be the PR which
is 25 =S.

It just seems really strange to me that lowering the number of max
hosts to 4000 and lower, stops the CPU from hitting the 100% mark and
dropping packets, yet when -g/--track-local-hosts is enabled, and the
hosts are only ~200, there is constant 100% cpu load and dropped
packets.

Thanks,

Nick
Quoting Gary Gatten <Ggatten [at] waddell>:

> What do you think your pps is? Can you confirm it with stats from
> your switch?
>
> Run top only on the ntop PID and enable thread view. There's
> another arg to top I can't recall right now, but it will help show
> you which thread is using the CPU. I'll see if I can find /
> remember this.
>
>
> -----Original Message-----
> From: ntop-bounces [at] listgateway
> [mailto:ntop-bounces [at] listgateway] On Behalf Of Nicholas
> Turner
> Sent: Thursday, June 17, 2010 1:13 PM
> To: ntop [at] listgateway
> Subject: Re: [Ntop] Track Local Hosts Abnormal CPU Usage
>
> I assume that would be the -n flag?
>
> Seems to have no effect on my operation with -x 4000 hosts, still same
> CPU usage hovering around 60-70%. Tried -n and --track-local-hosts,
> after 30 minutes of runtime, ntop has processed 1,798,000 packets, and
> libpcap has dropped just shy of 10,000,000 packets, and thinks it has
> only seen 197MB of traffic over this time (definitely been more, since
> 555% of packets have been dropped).
>
> So I would have to assume that the name resolution of IP's is not
> causing the problem, but thanks for the suggestion!
>
> On that note, I have also tried the -b flag to disable protocol
> decoding, but that did not seem to help my CPU usage.
>
> Nick
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop
>



_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


Ggatten at waddell

Jun 17, 2010, 1:33 PM

Post #6 of 6 (1056 views)
Permalink
Re: Track Local Hosts Abnormal CPU Usage [In reply to]

Each thread should have a unique PID. Then you can look at the log file and associate the PID with a ntop process / function. Or you can run ntop in gdb - it tells you most everything you'd want to know, and a bunch of stuff you don't!

I don't know why changing those settings would cause increased load. Perhaps that's not the only variable?

Maybe recompile and install into a temp or dev directory and run that version unchanged and see what happens?

I would suspect you'd be able to process at least 2x if not 4x that pps rate without much trouble. It will be interesting to see the resolution to this.

I use netflow exclusively so I can't help troubleshoot libpcap stuff much. But, from my experience DNS uses the most CPU - other than libpcap.

Maybe try running tcpdump and see how it's CPU and loss compares to ntop? Make sure and capture full packets and maybe throw in-vvv as well. if tcpdump captures "everything" (all data, unfiltered) with little loss and only 30% CPU - then that would be interesting.

G


-----Original Message-----
From: ntop-bounces [at] listgateway [mailto:ntop-bounces [at] listgateway] On Behalf Of Nicholas Turner
Sent: Thursday, June 17, 2010 3:12 PM
To: ntop [at] listgateway
Subject: Re: [Ntop] Track Local Hosts Abnormal CPU Usage

Both the switch and NTOP are reporting in the 5500pkt/s to 6500pkt/s,
so I think the fiber tap is working fine, NTOP seems to be seeing all
of the packets.

I tried running top -H (thread view) -p10440 (my ntop PID), but it
just shows me a bunch of lt-ntop threads... none showing too much
discernible difference that might help me (so maybe I havent got the
right arguments!) Infact the only difference seems to be the PR which
is 25 =S.

It just seems really strange to me that lowering the number of max
hosts to 4000 and lower, stops the CPU from hitting the 100% mark and
dropping packets, yet when -g/--track-local-hosts is enabled, and the
hosts are only ~200, there is constant 100% cpu load and dropped
packets.

Thanks,

Nick
Quoting Gary Gatten <Ggatten [at] waddell>:

> What do you think your pps is? Can you confirm it with stats from
> your switch?
>
> Run top only on the ntop PID and enable thread view. There's
> another arg to top I can't recall right now, but it will help show
> you which thread is using the CPU. I'll see if I can find /
> remember this.
>
>
> -----Original Message-----
> From: ntop-bounces [at] listgateway
> [mailto:ntop-bounces [at] listgateway] On Behalf Of Nicholas
> Turner
> Sent: Thursday, June 17, 2010 1:13 PM
> To: ntop [at] listgateway
> Subject: Re: [Ntop] Track Local Hosts Abnormal CPU Usage
>
> I assume that would be the -n flag?
>
> Seems to have no effect on my operation with -x 4000 hosts, still same
> CPU usage hovering around 60-70%. Tried -n and --track-local-hosts,
> after 30 minutes of runtime, ntop has processed 1,798,000 packets, and
> libpcap has dropped just shy of 10,000,000 packets, and thinks it has
> only seen 197MB of traffic over this time (definitely been more, since
> 555% of packets have been dropped).
>
> So I would have to assume that the name resolution of IP's is not
> causing the problem, but thanks for the suggestion!
>
> On that note, I have also tried the -b flag to disable protocol
> decoding, but that did not seem to help my CPU usage.
>
> Nick
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop
>



_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop

NTop users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.