Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NTop: Misc

stability of RRD and ntop (and nProbe)

 

 

NTop misc RSS feed   Index | Next | Previous | View Threaded


lahondes at sibio

Jun 14, 2011, 10:50 AM

Post #1 of 1 (328 views)
Permalink
stability of RRD and ntop (and nProbe)

Hi,

I have some trouble making ntop (RRD in fact) stable. I have 8 probes (all some pfflowd daemon on 7 different pfsense (freebsd) firewalls) that send some NetFlow informations. Ntop works quite well for a few days but then some trouble arises...

I first noticed that there is a huge number (more than 100,000) of directories under /var/lib/ntop/rrd/<name of interface>/NetFlow. Curiously it happens only with some pfsenses.

I've tried 4 different installs (under FreeBSD, 2 different installs, with UFS then ZFS / DragonFlyBSD (Hammer) / Linux (tuned ext3)). Each time, there were issues with huge number of files (notably under UFS), either the kernel would crash (ZFS/Hammer) or ntop/RRD would crash (UFS/Ext3). Finally, under Linux/Debian, I have followed all recommandation here about filesystem:

http://oss.oetiker.ch/rrdtool-trac/wiki/TuningRRD"]http://oss.oetiker.ch/rrdtool-trac/wiki/TuningRRD

That is:

Files System Level Optimization
  • Set the noatime and nodiratime option when mounting the ext3 file system. This will save updates to the directory blocks. (DavePlonka??)
  • Mount ext3 with data=writeback. This gives some performance boost at the expense of a possibility for old data to appear after a crash. Check man mount for details. (RyanKubica??)
  • Use a file system that indexes directories (ext3 with dir_index) and keep all rrd files in a single directory (DavePlonka??)

Yet, after a few days, my RRD graphs just disappeared (they do not update). When I relaunch ntop, they come back again.

I have checked in FAQ and forums and it seems that I am the only one with this problem.

I have two questions :
  • Have I done something wrong? Is it not some kind of well known problem with Ntop/NetFlow/RRD? (because I am nearly using it in a "out of the box" mode, just that I have 7 or 8 pfflowd probes, but I must not be the only one in this case).

  • Would nProbe, which has database backends, help me get rid of that huge number of files or just stabilize ntop in this case?

Many thanks !

Raynald
PS: I use ntop version v4.0.3

NTop misc RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.