Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NTop: Users

known double free bug in 4.0.3?

 

 

NTop users RSS feed   Index | Next | Previous | View Threaded


rick.jones2 at hp

Apr 11, 2011, 10:18 AM

Post #1 of 22 (2623 views)
Permalink
known double free bug in 4.0.3?

So, having temporarily worked-around the bit with the libraries not
being found, I left 4.0.3 running on an Ubuntu 10.10 system over the
weekend, receiving sFlow counter and flow samples from a switch. When I
got in this morning I saw an abort message:

*** glibc detected *** /usr/local/bin/ntop: double free or corruption
(fasttop): 0x00007f6db04e7990 ***
======= Backtrace: =========
/lib/libc.so.6(+0x774b6)[0x7f6dc0a584b6]
/lib/libc.so.6(cfree+0x73)[0x7f6dc0a5ec83]
/usr/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x7f6dc0fb5826]
/usr/lib/libntop-4.0.3.so(dequeueAddress+0x49a)[0x7f6dc0faa54a]
/lib/libpthread.so.0(+0x7971)[0x7f6dbfb91971]
/lib/libc.so.6(clone+0x6d)[0x7f6dc0ac792d]

is this a known thing, or is it new and requires a more formal bug
report?

rick jones
Attachments: dump.txt (6.63 KB)


deri at ntop

Apr 11, 2011, 11:51 AM

Post #2 of 22 (2566 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

Rick
this is a problem that from time to time shows up. I have analyzed the code a few times and not been able to spot this bug. I appreciate if you could help

Thanks Luca

On Apr 11, 2011, at 7:18 PM, Rick Jones wrote:

> So, having temporarily worked-around the bit with the libraries not
> being found, I left 4.0.3 running on an Ubuntu 10.10 system over the
> weekend, receiving sFlow counter and flow samples from a switch. When I
> got in this morning I saw an abort message:
>
> *** glibc detected *** /usr/local/bin/ntop: double free or corruption
> (fasttop): 0x00007f6db04e7990 ***
> ======= Backtrace: =========
> /lib/libc.so.6(+0x774b6)[0x7f6dc0a584b6]
> /lib/libc.so.6(cfree+0x73)[0x7f6dc0a5ec83]
> /usr/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x7f6dc0fb5826]
> /usr/lib/libntop-4.0.3.so(dequeueAddress+0x49a)[0x7f6dc0faa54a]
> /lib/libpthread.so.0(+0x7971)[0x7f6dbfb91971]
> /lib/libc.so.6(clone+0x6d)[0x7f6dc0ac792d]
>
> is this a known thing, or is it new and requires a more formal bug
> report?
>
> rick jones
> <dump.txt>_______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop

---
We can't solve problems by using the same kind of thinking we used when we created them - Albert Einstein

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


rick.jones2 at hp

Apr 11, 2011, 1:05 PM

Post #3 of 22 (2570 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

On Mon, 2011-04-11 at 20:51 +0200, Luca Deri wrote:
> Rick this is a problem that from time to time shows up. I have analyzed
> the code a few times and not been able to spot this bug. I appreciate
> if you could help

I will see what I can find while I'm blindly trying to feel my way
around the elephant (that old adage about the several blind men trying
to describe an elephant and each saying something completely different
based on what part of the elephant they were at :)

That will be while I'm poking around the sFlow and RRD areas - I am
anxious to figure-out if indeed I can get "pretty pictures" down to one
second granularity if I happen to have sFlow counter samples arriving at
that rate. The whole N days for interval stats and such heirarchy seems
quite well suited to my desires - which is to be able to look at what
was going-on down to a one second granularity going back oh three or
five days, but letting it aggregate after that.

rick

>
> Thanks Luca
>
> On Apr 11, 2011, at 7:18 PM, Rick Jones wrote:
>
> > So, having temporarily worked-around the bit with the libraries not
> > being found, I left 4.0.3 running on an Ubuntu 10.10 system over the
> > weekend, receiving sFlow counter and flow samples from a switch. When I
> > got in this morning I saw an abort message:
> >
> > *** glibc detected *** /usr/local/bin/ntop: double free or corruption
> > (fasttop): 0x00007f6db04e7990 ***
> > ======= Backtrace: =========
> > /lib/libc.so.6(+0x774b6)[0x7f6dc0a584b6]
> > /lib/libc.so.6(cfree+0x73)[0x7f6dc0a5ec83]
> > /usr/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x7f6dc0fb5826]
> > /usr/lib/libntop-4.0.3.so(dequeueAddress+0x49a)[0x7f6dc0faa54a]
> > /lib/libpthread.so.0(+0x7971)[0x7f6dbfb91971]
> > /lib/libc.so.6(clone+0x6d)[0x7f6dc0ac792d]
> >
> > is this a known thing, or is it new and requires a more formal bug
> > report?
> >
> > rick jones
> > <dump.txt>_______________________________________________
> > Ntop mailing list
> > Ntop [at] listgateway
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> ---
> We can't solve problems by using the same kind of thinking we used when we created them - Albert Einstein


_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


Ggatten at waddell

Apr 11, 2011, 1:23 PM

Post #4 of 22 (2575 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

Hmmm. I don't *think* ntop will give you the granularity nor history you speak of - not without major tweaking. I could be wrong, but 5 minute granularity is about all you can expect, in some cases 60 seconds. Going back "n days" will also prove problematic, unless you have lots of RAM / small numbers of hosts /etc and make them "sticky". Many "detail" counters age out after 24 hours.

You mentioned playing with rrd - MAYBE some knob turning will give you what you need, but I'm thinking not. One second granularity for several hours is hard enough, let alone several days. And, if I'm not mistaken sflow is a periodic sample anyway, and as accurate as the claims of sampled data may or may not be, I suspect it's not even close if you're looking second by second. Netflow won't even help much then.

You may want to consider nTop as the "big picture view" and something like Wireshark capturing "everything" you're interested in and saving to a new file every hour on the hour for 72 hours (or whatever). You'll need lot's of disk :) You can save some space by only capturing headers and skipping the data.

LMK what you come up with

G


-----Original Message-----
From: ntop-bounces [at] listgateway [mailto:ntop-bounces [at] listgateway] On Behalf Of Rick Jones
Sent: Monday, April 11, 2011 3:06 PM
To: Luca Deri
Cc: ntop [at] unipi
Subject: Re: [Ntop] known double free bug in 4.0.3?

On Mon, 2011-04-11 at 20:51 +0200, Luca Deri wrote:
> Rick this is a problem that from time to time shows up. I have analyzed
> the code a few times and not been able to spot this bug. I appreciate
> if you could help

I will see what I can find while I'm blindly trying to feel my way
around the elephant (that old adage about the several blind men trying
to describe an elephant and each saying something completely different
based on what part of the elephant they were at :)

That will be while I'm poking around the sFlow and RRD areas - I am
anxious to figure-out if indeed I can get "pretty pictures" down to one
second granularity if I happen to have sFlow counter samples arriving at
that rate. The whole N days for interval stats and such heirarchy seems
quite well suited to my desires - which is to be able to look at what
was going-on down to a one second granularity going back oh three or
five days, but letting it aggregate after that.

rick

>
> Thanks Luca
>
> On Apr 11, 2011, at 7:18 PM, Rick Jones wrote:
>
> > So, having temporarily worked-around the bit with the libraries not
> > being found, I left 4.0.3 running on an Ubuntu 10.10 system over the
> > weekend, receiving sFlow counter and flow samples from a switch. When I
> > got in this morning I saw an abort message:
> >
> > *** glibc detected *** /usr/local/bin/ntop: double free or corruption
> > (fasttop): 0x00007f6db04e7990 ***
> > ======= Backtrace: =========
> > /lib/libc.so.6(+0x774b6)[0x7f6dc0a584b6]
> > /lib/libc.so.6(cfree+0x73)[0x7f6dc0a5ec83]
> > /usr/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x7f6dc0fb5826]
> > /usr/lib/libntop-4.0.3.so(dequeueAddress+0x49a)[0x7f6dc0faa54a]
> > /lib/libpthread.so.0(+0x7971)[0x7f6dbfb91971]
> > /lib/libc.so.6(clone+0x6d)[0x7f6dc0ac792d]
> >
> > is this a known thing, or is it new and requires a more formal bug
> > report?
> >
> > rick jones
> > <dump.txt>_______________________________________________
> > Ntop mailing list
> > Ntop [at] listgateway
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> ---
> We can't solve problems by using the same kind of thinking we used when we created them - Albert Einstein


_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


rick.jones2 at hp

Apr 11, 2011, 1:37 PM

Post #5 of 22 (2570 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

On Mon, 2011-04-11 at 15:23 -0500, Gary Gatten wrote:
> Hmmm. I don't *think* ntop will give you the granularity nor history
> you speak of - not without major tweaking. I could be wrong, but 5
> minute granularity is about all you can expect, in some cases 60
> seconds. Going back "n days" will also prove problematic, unless you
> have lots of RAM / small numbers of hosts /etc and make them
> "sticky". Many "detail" counters age out after 24 hours.
>
> You mentioned playing with rrd - MAYBE some knob turning will give you
> what you need, but I'm thinking not. One second granularity for
> several hours is hard enough, let alone several days. And, if I'm not
> mistaken sflow is a periodic sample anyway, and as accurate as the
> claims of sampled data may or may not be, I suspect it's not even
> close if you're looking second by second. Netflow won't even help
> much then.

I'm not looking at flows second by second - I'm interested in counters
second by second - the per-interface counters on my switches. I don't
have nearly as great an interest in the flow samples and may not even be
configuring switches to send flow samples.

> You may want to consider nTop as the "big picture view" and something
> like Wireshark capturing "everything" you're interested in and saving
> to a new file every hour on the hour for 72 hours (or whatever).
> You'll need lot's of disk :) You can save some space by only
> capturing headers and skipping the data.

My current "plan B" is using sflowtool to dump counter samples, but then
I have to find the nice ways to turn that into pretty pictures. Again,
I'm not really interested in flow samples (or I guess at least I've not
learned I'm interested in flow samples :) just counter samples.

> LMK what you come up with

First thing I come-up with then is a question :) Well, perhaps more than
one :)

When I go the RRD plugin preferences page:

http://127.0.0.1:3000/plugins/rrdPlugin

There is an entry for "Dump Interval" and the web form allows me to set
it to "1" - what then am I actually storing? I was ass-u-me-ing the
interface stats would be dumped, and perhaps other things, based on the
"Data to Dump" check boxes further down. I wasn't sure though how that
might interact with sflow counter samples that do not arrive more often
than 5 or 10 or 20 seconds per switch port.

Similarly, I'm permitted to set "Throughput Granularity" to 1 second.

"Dump Hours" and "Dump Days" and "Dump Months" seemed like they would be
doing bidding similar to my desires wrt aggregating data for longer
term.

BTW, there appears to be a typo in the "RRDcached Path" description - it
reads in part "overloads ntop from RRD file update." I suspect that was
meant to be something like "offloads RRD RRD file update from ntop."

rick

>
> G
>
>
> -----Original Message-----
> From: ntop-bounces [at] listgateway [mailto:ntop-bounces [at] listgateway] On Behalf Of Rick Jones
> Sent: Monday, April 11, 2011 3:06 PM
> To: Luca Deri
> Cc: ntop [at] unipi
> Subject: Re: [Ntop] known double free bug in 4.0.3?
>
> On Mon, 2011-04-11 at 20:51 +0200, Luca Deri wrote:
> > Rick this is a problem that from time to time shows up. I have analyzed
> > the code a few times and not been able to spot this bug. I appreciate
> > if you could help
>
> I will see what I can find while I'm blindly trying to feel my way
> around the elephant (that old adage about the several blind men trying
> to describe an elephant and each saying something completely different
> based on what part of the elephant they were at :)
>
> That will be while I'm poking around the sFlow and RRD areas - I am
> anxious to figure-out if indeed I can get "pretty pictures" down to one
> second granularity if I happen to have sFlow counter samples arriving at
> that rate. The whole N days for interval stats and such heirarchy seems
> quite well suited to my desires - which is to be able to look at what
> was going-on down to a one second granularity going back oh three or
> five days, but letting it aggregate after that.
>
> rick
>
> >
> > Thanks Luca
> >
> > On Apr 11, 2011, at 7:18 PM, Rick Jones wrote:
> >
> > > So, having temporarily worked-around the bit with the libraries not
> > > being found, I left 4.0.3 running on an Ubuntu 10.10 system over the
> > > weekend, receiving sFlow counter and flow samples from a switch. When I
> > > got in this morning I saw an abort message:
> > >
> > > *** glibc detected *** /usr/local/bin/ntop: double free or corruption
> > > (fasttop): 0x00007f6db04e7990 ***
> > > ======= Backtrace: =========
> > > /lib/libc.so.6(+0x774b6)[0x7f6dc0a584b6]
> > > /lib/libc.so.6(cfree+0x73)[0x7f6dc0a5ec83]
> > > /usr/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x7f6dc0fb5826]
> > > /usr/lib/libntop-4.0.3.so(dequeueAddress+0x49a)[0x7f6dc0faa54a]
> > > /lib/libpthread.so.0(+0x7971)[0x7f6dbfb91971]
> > > /lib/libc.so.6(clone+0x6d)[0x7f6dc0ac792d]
> > >
> > > is this a known thing, or is it new and requires a more formal bug
> > > report?
> > >
> > > rick jones
> > > <dump.txt>_______________________________________________
> > > Ntop mailing list
> > > Ntop [at] listgateway
> > > http://listgateway.unipi.it/mailman/listinfo/ntop
> >
> > ---
> > We can't solve problems by using the same kind of thinking we used when we created them - Albert Einstein
>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop


_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


Ggatten at waddell

Apr 11, 2011, 1:50 PM

Post #6 of 22 (2572 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

I may have to read through this thread again, I'm confused. To my knowledge snmp and/or rmon is the only way to get interface "counters". Netflow, sflow, etc give you flow/conversation details.

----- Original Message -----
From: Rick Jones [mailto:rick.jones2 [at] hp]
Sent: Monday, April 11, 2011 03:37 PM
To: ntop [at] unipi <ntop [at] unipi>
Subject: Re: [Ntop] known double free bug in 4.0.3?

On Mon, 2011-04-11 at 15:23 -0500, Gary Gatten wrote:
> Hmmm. I don't *think* ntop will give you the granularity nor history
> you speak of - not without major tweaking. I could be wrong, but 5
> minute granularity is about all you can expect, in some cases 60
> seconds. Going back "n days" will also prove problematic, unless you
> have lots of RAM / small numbers of hosts /etc and make them
> "sticky". Many "detail" counters age out after 24 hours.
>
> You mentioned playing with rrd - MAYBE some knob turning will give you
> what you need, but I'm thinking not. One second granularity for
> several hours is hard enough, let alone several days. And, if I'm not
> mistaken sflow is a periodic sample anyway, and as accurate as the
> claims of sampled data may or may not be, I suspect it's not even
> close if you're looking second by second. Netflow won't even help
> much then.

I'm not looking at flows second by second - I'm interested in counters
second by second - the per-interface counters on my switches. I don't
have nearly as great an interest in the flow samples and may not even be
configuring switches to send flow samples.

> You may want to consider nTop as the "big picture view" and something
> like Wireshark capturing "everything" you're interested in and saving
> to a new file every hour on the hour for 72 hours (or whatever).
> You'll need lot's of disk :) You can save some space by only
> capturing headers and skipping the data.

My current "plan B" is using sflowtool to dump counter samples, but then
I have to find the nice ways to turn that into pretty pictures. Again,
I'm not really interested in flow samples (or I guess at least I've not
learned I'm interested in flow samples :) just counter samples.

> LMK what you come up with

First thing I come-up with then is a question :) Well, perhaps more than
one :)

When I go the RRD plugin preferences page:

http://127.0.0.1:3000/plugins/rrdPlugin

There is an entry for "Dump Interval" and the web form allows me to set
it to "1" - what then am I actually storing? I was ass-u-me-ing the
interface stats would be dumped, and perhaps other things, based on the
"Data to Dump" check boxes further down. I wasn't sure though how that
might interact with sflow counter samples that do not arrive more often
than 5 or 10 or 20 seconds per switch port.

Similarly, I'm permitted to set "Throughput Granularity" to 1 second.

"Dump Hours" and "Dump Days" and "Dump Months" seemed like they would be
doing bidding similar to my desires wrt aggregating data for longer
term.

BTW, there appears to be a typo in the "RRDcached Path" description - it
reads in part "overloads ntop from RRD file update." I suspect that was
meant to be something like "offloads RRD RRD file update from ntop."

rick

>
> G
>
>
> -----Original Message-----
> From: ntop-bounces [at] listgateway [mailto:ntop-bounces [at] listgateway] On Behalf Of Rick Jones
> Sent: Monday, April 11, 2011 3:06 PM
> To: Luca Deri
> Cc: ntop [at] unipi
> Subject: Re: [Ntop] known double free bug in 4.0.3?
>
> On Mon, 2011-04-11 at 20:51 +0200, Luca Deri wrote:
> > Rick this is a problem that from time to time shows up. I have analyzed
> > the code a few times and not been able to spot this bug. I appreciate
> > if you could help
>
> I will see what I can find while I'm blindly trying to feel my way
> around the elephant (that old adage about the several blind men trying
> to describe an elephant and each saying something completely different
> based on what part of the elephant they were at :)
>
> That will be while I'm poking around the sFlow and RRD areas - I am
> anxious to figure-out if indeed I can get "pretty pictures" down to one
> second granularity if I happen to have sFlow counter samples arriving at
> that rate. The whole N days for interval stats and such heirarchy seems
> quite well suited to my desires - which is to be able to look at what
> was going-on down to a one second granularity going back oh three or
> five days, but letting it aggregate after that.
>
> rick
>
> >
> > Thanks Luca
> >
> > On Apr 11, 2011, at 7:18 PM, Rick Jones wrote:
> >
> > > So, having temporarily worked-around the bit with the libraries not
> > > being found, I left 4.0.3 running on an Ubuntu 10.10 system over the
> > > weekend, receiving sFlow counter and flow samples from a switch. When I
> > > got in this morning I saw an abort message:
> > >
> > > *** glibc detected *** /usr/local/bin/ntop: double free or corruption
> > > (fasttop): 0x00007f6db04e7990 ***
> > > ======= Backtrace: =========
> > > /lib/libc.so.6(+0x774b6)[0x7f6dc0a584b6]
> > > /lib/libc.so.6(cfree+0x73)[0x7f6dc0a5ec83]
> > > /usr/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x7f6dc0fb5826]
> > > /usr/lib/libntop-4.0.3.so(dequeueAddress+0x49a)[0x7f6dc0faa54a]
> > > /lib/libpthread.so.0(+0x7971)[0x7f6dbfb91971]
> > > /lib/libc.so.6(clone+0x6d)[0x7f6dc0ac792d]
> > >
> > > is this a known thing, or is it new and requires a more formal bug
> > > report?
> > >
> > > rick jones
> > > <dump.txt>_______________________________________________
> > > Ntop mailing list
> > > Ntop [at] listgateway
> > > http://listgateway.unipi.it/mailman/listinfo/ntop
> >
> > ---
> > We can't solve problems by using the same kind of thinking we used when we created them - Albert Einstein
>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop


_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


rick.jones2 at hp

Apr 11, 2011, 2:11 PM

Post #7 of 22 (2568 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

On Mon, 2011-04-11 at 15:50 -0500, Gary Gatten wrote:
> I may have to read through this thread again, I'm confused. To my
> knowledge snmp and/or rmon is the only way to get interface
> "counters". Netflow, sflow, etc give you flow/conversation details.

It may help to think of sFlow as "Shimmer" - it is both a floor wax
*and* a dessert topping. An sflow agent can be configured to send flow
samples (similar to netflow) or counter samples to an sflow collector,
and it does appear that ntop can receive and do something useful with
sflow counter samples.

The Very Nice Thing about an sFlow counter sample is one gets all the
counters in one shot - so instead of having to query each stat for an
interface one at a time via SNMP, they all "just arrive" together in the
same sFlow PDU. In fact, multiple counter samples, each from multiple
interfaces, can be carried in the same sflow PDU making it even that
much more efficient than querying via SNMP.

rick


_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


Ggatten at waddell

Apr 11, 2011, 2:17 PM

Post #8 of 22 (2568 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

Ah, I did not know sflow could do such things. I wonder if it hooks into the MIBs similar to RMON. Interesting. Yeah, even with extended snmp gets its not all that effecient.

----- Original Message -----
From: Rick Jones [mailto:rick.jones2 [at] hp]
Sent: Monday, April 11, 2011 04:11 PM
To: Gary Gatten
Cc: 'ntop [at] unipi' <ntop [at] unipi>
Subject: Re: [Ntop] known double free bug in 4.0.3?

On Mon, 2011-04-11 at 15:50 -0500, Gary Gatten wrote:
> I may have to read through this thread again, I'm confused. To my
> knowledge snmp and/or rmon is the only way to get interface
> "counters". Netflow, sflow, etc give you flow/conversation details.

It may help to think of sFlow as "Shimmer" - it is both a floor wax
*and* a dessert topping. An sflow agent can be configured to send flow
samples (similar to netflow) or counter samples to an sflow collector,
and it does appear that ntop can receive and do something useful with
sflow counter samples.

The Very Nice Thing about an sFlow counter sample is one gets all the
counters in one shot - so instead of having to query each stat for an
interface one at a time via SNMP, they all "just arrive" together in the
same sFlow PDU. In fact, multiple counter samples, each from multiple
interfaces, can be carried in the same sflow PDU making it even that
much more efficient than querying via SNMP.

rick







<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


rick.jones2 at hp

Apr 11, 2011, 2:17 PM

Post #9 of 22 (2575 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

On Mon, 2011-04-11 at 14:11 -0700, Rick Jones wrote:
> On Mon, 2011-04-11 at 15:50 -0500, Gary Gatten wrote:
> > I may have to read through this thread again, I'm confused. To my
> > knowledge snmp and/or rmon is the only way to get interface
> > "counters". Netflow, sflow, etc give you flow/conversation details.
>
> It may help to think of sFlow as "Shimmer" - it is both a floor wax
> *and* a dessert topping. An sflow agent can be configured to send flow
> samples (similar to netflow) or counter samples to an sflow collector,

actually that should be "and/or" - both can be happening at the same
time. I'd give an example of tcpdump output, but "in another thread"
I'm involved in issues with the correct decode of sflow PDUs :) I can
though show you some incomplete output:

16:47:41.409631 IP (tos 0x0, ttl 64, id 49088, offset 0, flags [none],
proto UDP (17), length 1348)
the-switch.54321 > z400.sflow: [udp sum ok] sFlowv5, IPv4 agent
the-switch, agent-id 0, seqnum 6903, uptime 2437130, samples 7, length
1320
flow sample (1), length 208,
flow sample (1), length 148,
flow sample (1), length 208,
counter sample (2), length 168,
counter sample (2), length 168,
counter sample (2), length 168,
counter sample (2), length 168,


> and it does appear that ntop can receive and do something useful with
> sflow counter samples.
>
> The Very Nice Thing about an sFlow counter sample is one gets all the
> counters in one shot - so instead of having to query each stat for an
> interface one at a time via SNMP, they all "just arrive" together in the
> same sFlow PDU. In fact, multiple counter samples, each from multiple
> interfaces, can be carried in the same sflow PDU making it even that
> much more efficient than querying via SNMP.
>
> rick
>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop


_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


schrodinger at konundrum

Apr 12, 2011, 2:01 AM

Post #10 of 22 (2565 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

Hi,

I have this exact same problem, "double free or corruption". I have
compiled Ntop with debugging and electric fence and the problem still
persists. Usually Ntop can't last more than a few hours at a time :( I
am using Ntop to collect Netflow from different sources via thge native
Netflow plugin.

I had compiled Ntop with additional debugging to try and locate the
problem.

[user [at] hos ntop-4.0.3]# export CFLAGS+=-g
[user [at] hos ntop-4.0.3]# echo $CFLAGS
-g
[user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2

===================================================================

This is your ntop 4.0.3 configuration:

Host System Type : x86_64-redhat-linux-gnu
Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
Defines : -DHAVE_CONFIG_H
Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
Include path :
System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
Locale : /usr/lib/locale
Memory Debug : 3

External packages:

LBL pcap .h : standard system headers
LBL pcap library : standard system libraries
GNU gdbm .h : standard system headers
GNU gdbm library : standard system libraries
zlib .h : standard system headers
zlib library : standard system libraries
openSSL .h : standard system headers
openSSL library : standard system libraries

Install directories:

Default prefix: /usr/local
Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)

Data files are in /usr/local/ntop-4.0.3/share/ntop
Config files are in /usr/local/ntop-4.0.3/etc/ntop
Run directory is /usr/local/ntop-4.0.3/var/ntop
Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
Database files are in /usr/local/ntop-4.0.3/var/ntop

-------------------------------------------------------------------

Here are some logs then that I was able to glean from the system log and/or gdb.

127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
*** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
aab41ede00 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2abf39c0430f]
/lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
/usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
/usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
/lib64/libpthread.so.0[0x2abf3974473d]
/lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]

While running the main Ntop process in gdb:

Detaching after fork from child process 22963.
Detaching after fork from child process 22964.
Detaching after fork from child process 22965.
Detaching after fork from child process 22966.
Detaching after fork from child process 22968.
Detaching after fork from child process 22969.
Detaching after fork from child process 22970.
[...]
Detaching after fork from child process 22971.
*** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2aaaab8b930f]
/lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
/usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
/lib64/libpthread.so.0[0x2aaaab3f973d]
/lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
======= Memory map: ========
00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
00611000-01892000 rw-p 00611000 00:00 0 [heap]
40000000-40001000 ---p 40000000 00:00 0
40001000-40a01000 rw-p 40001000 00:00 0
40a01000-40a02000 ---p 40a01000 00:00 0
40a02000-41402000 rw-p 40a02000 00:00 0
41402000-41403000 ---p 41402000 00:00 0
41403000-41e03000 rw-p 41403000 00:00 0
41e03000-41e04000 ---p 41e03000 00:00 0
41e04000-42804000 rw-p 41e04000 00:00 0
42804000-42805000 ---p 42804000 00:00 0
42805000-43205000 rw-p 42805000 00:00 0
43205000-43206000 ---p 43205000 00:00 0
43206000-43c06000 rw-p 43206000 00:00 0
43c06000-43c07000 ---p 43c06000 00:00 0
[...]
2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x41e02940 (LWP 22660)]
0x00002aaaab877265 in raise () from /lib64/libc.so.6
(gdb)
(gdb)

(gdb) bt
#0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
#1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
#2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
#3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
#4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
#5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
#6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
#7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
(gdb)

I am happy to provide any more information you might need or perform any
additional debugging tasks that would help track this issue down.

Thanks,
Conor.
--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc


Ggatten at waddell

Apr 12, 2011, 5:28 AM

Post #11 of 22 (2548 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

If it dies so often, perhaps try running it with a bunch of stuff disabled; name lookups, decoders, rrd (as much as possible), etc. and see if that makes a difference.

Have you tried running in gdb? I found many of my issues there, except once when it would die every hour running "normal", but running in gdb it would NOT crash! Typical :). The '-k (or -K?)' MAY have had something to do with that.

----- Original Message -----
From: Schrodinger [mailto:schrodinger [at] konundrum]
Sent: Tuesday, April 12, 2011 04:01 AM
To: ntop [at] listgateway <ntop [at] listgateway>
Subject: Re: [Ntop] known double free bug in 4.0.3?

Hi,

I have this exact same problem, "double free or corruption". I have
compiled Ntop with debugging and electric fence and the problem still
persists. Usually Ntop can't last more than a few hours at a time :( I
am using Ntop to collect Netflow from different sources via thge native
Netflow plugin.

I had compiled Ntop with additional debugging to try and locate the
problem.

[user [at] hos ntop-4.0.3]# export CFLAGS+=-g
[user [at] hos ntop-4.0.3]# echo $CFLAGS
-g
[user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2

===================================================================

This is your ntop 4.0.3 configuration:

Host System Type : x86_64-redhat-linux-gnu
Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
Defines : -DHAVE_CONFIG_H
Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
Include path :
System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
Locale : /usr/lib/locale
Memory Debug : 3

External packages:

LBL pcap .h : standard system headers
LBL pcap library : standard system libraries
GNU gdbm .h : standard system headers
GNU gdbm library : standard system libraries
zlib .h : standard system headers
zlib library : standard system libraries
openSSL .h : standard system headers
openSSL library : standard system libraries

Install directories:

Default prefix: /usr/local
Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)

Data files are in /usr/local/ntop-4.0.3/share/ntop
Config files are in /usr/local/ntop-4.0.3/etc/ntop
Run directory is /usr/local/ntop-4.0.3/var/ntop
Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
Database files are in /usr/local/ntop-4.0.3/var/ntop

-------------------------------------------------------------------

Here are some logs then that I was able to glean from the system log and/or gdb.

127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
*** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
aab41ede00 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2abf39c0430f]
/lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
/usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
/usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
/lib64/libpthread.so.0[0x2abf3974473d]
/lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]

While running the main Ntop process in gdb:

Detaching after fork from child process 22963.
Detaching after fork from child process 22964.
Detaching after fork from child process 22965.
Detaching after fork from child process 22966.
Detaching after fork from child process 22968.
Detaching after fork from child process 22969.
Detaching after fork from child process 22970.
[...]
Detaching after fork from child process 22971.
*** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2aaaab8b930f]
/lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
/usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
/lib64/libpthread.so.0[0x2aaaab3f973d]
/lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
======= Memory map: ========
00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
00611000-01892000 rw-p 00611000 00:00 0 [heap]
40000000-40001000 ---p 40000000 00:00 0
40001000-40a01000 rw-p 40001000 00:00 0
40a01000-40a02000 ---p 40a01000 00:00 0
40a02000-41402000 rw-p 40a02000 00:00 0
41402000-41403000 ---p 41402000 00:00 0
41403000-41e03000 rw-p 41403000 00:00 0
41e03000-41e04000 ---p 41e03000 00:00 0
41e04000-42804000 rw-p 41e04000 00:00 0
42804000-42805000 ---p 42804000 00:00 0
42805000-43205000 rw-p 42805000 00:00 0
43205000-43206000 ---p 43205000 00:00 0
43206000-43c06000 rw-p 43206000 00:00 0
43c06000-43c07000 ---p 43c06000 00:00 0
[...]
2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x41e02940 (LWP 22660)]
0x00002aaaab877265 in raise () from /lib64/libc.so.6
(gdb)
(gdb)

(gdb) bt
#0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
#1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
#2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
#3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
#4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
#5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
#6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
#7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
(gdb)

I am happy to provide any more information you might need or perform any
additional debugging tasks that would help track this issue down.

Thanks,
Conor.
--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


Ggatten at waddell

Apr 12, 2011, 5:28 AM

Post #12 of 22 (2547 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

If it dies so often, perhaps try running it with a bunch of stuff disabled; name lookups, decoders, rrd (as much as possible), etc. and see if that makes a difference.

Have you tried running in gdb? I found many of my issues there, except once when it would die every hour running "normal", but running in gdb it would NOT crash! Typical :). The '-k (or -K?)' MAY have had something to do with that.

----- Original Message -----
From: Schrodinger [mailto:schrodinger [at] konundrum]
Sent: Tuesday, April 12, 2011 04:01 AM
To: ntop [at] listgateway <ntop [at] listgateway>
Subject: Re: [Ntop] known double free bug in 4.0.3?

Hi,

I have this exact same problem, "double free or corruption". I have
compiled Ntop with debugging and electric fence and the problem still
persists. Usually Ntop can't last more than a few hours at a time :( I
am using Ntop to collect Netflow from different sources via thge native
Netflow plugin.

I had compiled Ntop with additional debugging to try and locate the
problem.

[user [at] hos ntop-4.0.3]# export CFLAGS+=-g
[user [at] hos ntop-4.0.3]# echo $CFLAGS
-g
[user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2

===================================================================

This is your ntop 4.0.3 configuration:

Host System Type : x86_64-redhat-linux-gnu
Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
Defines : -DHAVE_CONFIG_H
Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
Include path :
System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
Locale : /usr/lib/locale
Memory Debug : 3

External packages:

LBL pcap .h : standard system headers
LBL pcap library : standard system libraries
GNU gdbm .h : standard system headers
GNU gdbm library : standard system libraries
zlib .h : standard system headers
zlib library : standard system libraries
openSSL .h : standard system headers
openSSL library : standard system libraries

Install directories:

Default prefix: /usr/local
Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)

Data files are in /usr/local/ntop-4.0.3/share/ntop
Config files are in /usr/local/ntop-4.0.3/etc/ntop
Run directory is /usr/local/ntop-4.0.3/var/ntop
Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
Database files are in /usr/local/ntop-4.0.3/var/ntop

-------------------------------------------------------------------

Here are some logs then that I was able to glean from the system log and/or gdb.

127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
*** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
aab41ede00 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2abf39c0430f]
/lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
/usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
/usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
/lib64/libpthread.so.0[0x2abf3974473d]
/lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]

While running the main Ntop process in gdb:

Detaching after fork from child process 22963.
Detaching after fork from child process 22964.
Detaching after fork from child process 22965.
Detaching after fork from child process 22966.
Detaching after fork from child process 22968.
Detaching after fork from child process 22969.
Detaching after fork from child process 22970.
[...]
Detaching after fork from child process 22971.
*** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2aaaab8b930f]
/lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
/usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
/lib64/libpthread.so.0[0x2aaaab3f973d]
/lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
======= Memory map: ========
00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
00611000-01892000 rw-p 00611000 00:00 0 [heap]
40000000-40001000 ---p 40000000 00:00 0
40001000-40a01000 rw-p 40001000 00:00 0
40a01000-40a02000 ---p 40a01000 00:00 0
40a02000-41402000 rw-p 40a02000 00:00 0
41402000-41403000 ---p 41402000 00:00 0
41403000-41e03000 rw-p 41403000 00:00 0
41e03000-41e04000 ---p 41e03000 00:00 0
41e04000-42804000 rw-p 41e04000 00:00 0
42804000-42805000 ---p 42804000 00:00 0
42805000-43205000 rw-p 42805000 00:00 0
43205000-43206000 ---p 43205000 00:00 0
43206000-43c06000 rw-p 43206000 00:00 0
43c06000-43c07000 ---p 43c06000 00:00 0
[...]
2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x41e02940 (LWP 22660)]
0x00002aaaab877265 in raise () from /lib64/libc.so.6
(gdb)
(gdb)

(gdb) bt
#0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
#1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
#2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
#3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
#4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
#5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
#6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
#7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
(gdb)

I am happy to provide any more information you might need or perform any
additional debugging tasks that would help track this issue down.

Thanks,
Conor.
--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


schrodinger at konundrum

Apr 12, 2011, 6:23 AM

Post #13 of 22 (2536 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

On 2011/04/12 07:28, Gary Gatten wrote:
> If it dies so often, perhaps try running it with a bunch of stuff disabled; name lookups, decoders, rrd (as much as possible), etc. and see if that makes a difference.

Gary,

Error messages below are from Ntop running normally, not in gdb and also
running in gdb + gdb backtrace.

"While running the main Ntop process in gdb"

Conor.

>
> Have you tried running in gdb? I found many of my issues there, except once when it would die every hour running "normal", but running in gdb it would NOT crash! Typical :). The '-k (or -K?)' MAY have had something to do with that.
>
> ----- Original Message -----
> From: Schrodinger [mailto:schrodinger [at] konundrum]
> Sent: Tuesday, April 12, 2011 04:01 AM
> To: ntop [at] listgateway <ntop [at] listgateway>
> Subject: Re: [Ntop] known double free bug in 4.0.3?
>
> Hi,
>
> I have this exact same problem, "double free or corruption". I have
> compiled Ntop with debugging and electric fence and the problem still
> persists. Usually Ntop can't last more than a few hours at a time :( I
> am using Ntop to collect Netflow from different sources via thge native
> Netflow plugin.
>
> I had compiled Ntop with additional debugging to try and locate the
> problem.
>
> [user [at] hos ntop-4.0.3]# export CFLAGS+=-g
> [user [at] hos ntop-4.0.3]# echo $CFLAGS
> -g
> [user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2
>
> ===================================================================
>
> This is your ntop 4.0.3 configuration:
>
> Host System Type : x86_64-redhat-linux-gnu
> Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
> Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
> Defines : -DHAVE_CONFIG_H
> Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
> Include path :
> System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
> Locale : /usr/lib/locale
> Memory Debug : 3
>
> External packages:
>
> LBL pcap .h : standard system headers
> LBL pcap library : standard system libraries
> GNU gdbm .h : standard system headers
> GNU gdbm library : standard system libraries
> zlib .h : standard system headers
> zlib library : standard system libraries
> openSSL .h : standard system headers
> openSSL library : standard system libraries
>
> Install directories:
>
> Default prefix: /usr/local
> Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)
>
> Data files are in /usr/local/ntop-4.0.3/share/ntop
> Config files are in /usr/local/ntop-4.0.3/etc/ntop
> Run directory is /usr/local/ntop-4.0.3/var/ntop
> Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
> Database files are in /usr/local/ntop-4.0.3/var/ntop
>
> -------------------------------------------------------------------
>
> Here are some logs then that I was able to glean from the system log and/or gdb.
>
> 127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
> *** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
> aab41ede00 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x2abf39c0430f]
> /lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
> /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
> /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
> /lib64/libpthread.so.0[0x2abf3974473d]
> /lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]
>
> While running the main Ntop process in gdb:
>
> Detaching after fork from child process 22963.
> Detaching after fork from child process 22964.
> Detaching after fork from child process 22965.
> Detaching after fork from child process 22966.
> Detaching after fork from child process 22968.
> Detaching after fork from child process 22969.
> Detaching after fork from child process 22970.
> [...]
> Detaching after fork from child process 22971.
> *** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x2aaaab8b930f]
> /lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
> /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
> /lib64/libpthread.so.0[0x2aaaab3f973d]
> /lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
> ======= Memory map: ========
> 00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> 00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> 00611000-01892000 rw-p 00611000 00:00 0 [heap]
> 40000000-40001000 ---p 40000000 00:00 0
> 40001000-40a01000 rw-p 40001000 00:00 0
> 40a01000-40a02000 ---p 40a01000 00:00 0
> 40a02000-41402000 rw-p 40a02000 00:00 0
> 41402000-41403000 ---p 41402000 00:00 0
> 41403000-41e03000 rw-p 41403000 00:00 0
> 41e03000-41e04000 ---p 41e03000 00:00 0
> 41e04000-42804000 rw-p 41e04000 00:00 0
> 42804000-42805000 ---p 42804000 00:00 0
> 42805000-43205000 rw-p 42805000 00:00 0
> 43205000-43206000 ---p 43205000 00:00 0
> 43206000-43c06000 rw-p 43206000 00:00 0
> 43c06000-43c07000 ---p 43c06000 00:00 0
> [...]
> 2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
> 2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
> 2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
> 2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
> 2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
> 2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0x41e02940 (LWP 22660)]
> 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> (gdb)
> (gdb)
>
> (gdb) bt
> #0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> #1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
> #2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
> #3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
> #4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
> #5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
> #6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
> #7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
> (gdb)
>
> I am happy to provide any more information you might need or perform any
> additional debugging tasks that would help track this issue down.
>
> Thanks,
> Conor.
> --
> +---------------------------------------------------------------+
> It was a new day yesterday, but it's an old day now.
> MSN: schro5 [at] hotmail
> ICQ: 112562229
> GPG: http://www.konundrum.org/schro.asc
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop

--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc


schrodinger at konundrum

Apr 12, 2011, 6:34 AM

Post #14 of 22 (2539 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

Some additional logs I forgot to add:

Nov 11 14:29:50 mgmt ntop[11594]: LEAK: ERROR: free() - Ptr 0x2aaabc257b60 NOT allocated [@util.c:5893]
Nov 11 14:29:50 mgmt ntop[11594]: LEAK: ERROR: free() - Ptr 0x2aaabc5a83d0 NOT allocated [@util.c:5893]
Nov 11 14:29:50 mgmt ntop[11594]: LEAK: ERROR: free() - Ptr 0x2aaabc156570 NOT allocated [@util.c:5893]
Nov 11 14:29:51 mgmt ntop[11594]: LEAK: ERROR: free() - Ptr 0x2aaabc9a6060 NOT allocated [@util.c:5893]
Nov 11 14:29:51 mgmt ntop[11594]: LEAK: ERROR: free() - Ptr 0x2aaabc59ebd0 NOT allocated [@util.c:5893]
Nov 11 14:29:51 mgmt ntop[11594]: LEAK: ERROR: free() - Ptr 0x2aaabc0b9ac0 NOT allocated [@util.c:5893]
Nov 11 14:29:54 mgmt ntop[11594]: LEAK: ERROR: free() - Ptr 0x2aaabcab1be0 NOT allocated [@util.c:5893]
Nov 11 14:29:58 mgmt ntop[11594]: LEAK: ERROR: free() - Ptr 0x2aaabc662f40 NOT allocated [@util.c:5893]
Nov 11 14:29:59 mgmt ntop[11594]: LEAK: ERROR: free() - Ptr 0x2aaabc4d7cb0 NOT allocated [@util.c:5893]
Nov 11 14:30:10 mgmt ntop[11594]: LEAK: ERROR: free() - Ptr 0x2aaabc7231b0 NOT allocated [@util.c:5893]

Conor.

On 2011/04/12 10:01, Schrodinger wrote:
> Hi,
>
> I have this exact same problem, "double free or corruption". I have
> compiled Ntop with debugging and electric fence and the problem still
> persists. Usually Ntop can't last more than a few hours at a time :( I
> am using Ntop to collect Netflow from different sources via thge native
> Netflow plugin.
>
> I had compiled Ntop with additional debugging to try and locate the
> problem.
>
> [user [at] hos ntop-4.0.3]# export CFLAGS+=-g
> [user [at] hos ntop-4.0.3]# echo $CFLAGS
> -g
> [user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2
>
> ===================================================================
>
> This is your ntop 4.0.3 configuration:
>
> Host System Type : x86_64-redhat-linux-gnu
> Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
> Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
> Defines : -DHAVE_CONFIG_H
> Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
> Include path :
> System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
> Locale : /usr/lib/locale
> Memory Debug : 3
>
> External packages:
>
> LBL pcap .h : standard system headers
> LBL pcap library : standard system libraries
> GNU gdbm .h : standard system headers
> GNU gdbm library : standard system libraries
> zlib .h : standard system headers
> zlib library : standard system libraries
> openSSL .h : standard system headers
> openSSL library : standard system libraries
>
> Install directories:
>
> Default prefix: /usr/local
> Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)
>
> Data files are in /usr/local/ntop-4.0.3/share/ntop
> Config files are in /usr/local/ntop-4.0.3/etc/ntop
> Run directory is /usr/local/ntop-4.0.3/var/ntop
> Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
> Database files are in /usr/local/ntop-4.0.3/var/ntop
>
> -------------------------------------------------------------------
>
> Here are some logs then that I was able to glean from the system log and/or gdb.
>
> 127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
> *** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
> aab41ede00 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x2abf39c0430f]
> /lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
> /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
> /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
> /lib64/libpthread.so.0[0x2abf3974473d]
> /lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]
>
> While running the main Ntop process in gdb:
>
> Detaching after fork from child process 22963.
> Detaching after fork from child process 22964.
> Detaching after fork from child process 22965.
> Detaching after fork from child process 22966.
> Detaching after fork from child process 22968.
> Detaching after fork from child process 22969.
> Detaching after fork from child process 22970.
> [...]
> Detaching after fork from child process 22971.
> *** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x2aaaab8b930f]
> /lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
> /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
> /lib64/libpthread.so.0[0x2aaaab3f973d]
> /lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
> ======= Memory map: ========
> 00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> 00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> 00611000-01892000 rw-p 00611000 00:00 0 [heap]
> 40000000-40001000 ---p 40000000 00:00 0
> 40001000-40a01000 rw-p 40001000 00:00 0
> 40a01000-40a02000 ---p 40a01000 00:00 0
> 40a02000-41402000 rw-p 40a02000 00:00 0
> 41402000-41403000 ---p 41402000 00:00 0
> 41403000-41e03000 rw-p 41403000 00:00 0
> 41e03000-41e04000 ---p 41e03000 00:00 0
> 41e04000-42804000 rw-p 41e04000 00:00 0
> 42804000-42805000 ---p 42804000 00:00 0
> 42805000-43205000 rw-p 42805000 00:00 0
> 43205000-43206000 ---p 43205000 00:00 0
> 43206000-43c06000 rw-p 43206000 00:00 0
> 43c06000-43c07000 ---p 43c06000 00:00 0
> [...]
> 2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
> 2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
> 2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
> 2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
> 2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
> 2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0x41e02940 (LWP 22660)]
> 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> (gdb)
> (gdb)
>
> (gdb) bt
> #0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> #1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
> #2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
> #3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
> #4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
> #5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
> #6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
> #7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
> (gdb)
>
> I am happy to provide any more information you might need or perform any
> additional debugging tasks that would help track this issue down.
>
> Thanks,
> Conor.
> --
> +---------------------------------------------------------------+
> It was a new day yesterday, but it's an old day now.
> MSN: schro5 [at] hotmail
> ICQ: 112562229
> GPG: http://www.konundrum.org/schro.asc



> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop


--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc


Ggatten at waddell

Apr 12, 2011, 6:44 AM

Post #15 of 22 (2538 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

Certainly not an expert, but I see text like "dequeueAddress" and think name/ip lookups. Disable this (-z?) and see how things go. I've had issues with the resolution process that disabling fixed. Maybe some of those same issues aren't fully patched and are impacting you?

----- Original Message -----
From: Schrodinger [mailto:schrodinger [at] konundrum]
Sent: Tuesday, April 12, 2011 08:23 AM
To: ntop [at] listgateway <ntop [at] listgateway>
Subject: Re: [Ntop] known double free bug in 4.0.3?

On 2011/04/12 07:28, Gary Gatten wrote:
> If it dies so often, perhaps try running it with a bunch of stuff disabled; name lookups, decoders, rrd (as much as possible), etc. and see if that makes a difference.

Gary,

Error messages below are from Ntop running normally, not in gdb and also
running in gdb + gdb backtrace.

"While running the main Ntop process in gdb"

Conor.

>
> Have you tried running in gdb? I found many of my issues there, except once when it would die every hour running "normal", but running in gdb it would NOT crash! Typical :). The '-k (or -K?)' MAY have had something to do with that.
>
> ----- Original Message -----
> From: Schrodinger [mailto:schrodinger [at] konundrum]
> Sent: Tuesday, April 12, 2011 04:01 AM
> To: ntop [at] listgateway <ntop [at] listgateway>
> Subject: Re: [Ntop] known double free bug in 4.0.3?
>
> Hi,
>
> I have this exact same problem, "double free or corruption". I have
> compiled Ntop with debugging and electric fence and the problem still
> persists. Usually Ntop can't last more than a few hours at a time :( I
> am using Ntop to collect Netflow from different sources via thge native
> Netflow plugin.
>
> I had compiled Ntop with additional debugging to try and locate the
> problem.
>
> [user [at] hos ntop-4.0.3]# export CFLAGS+=-g
> [user [at] hos ntop-4.0.3]# echo $CFLAGS
> -g
> [user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2
>
> ===================================================================
>
> This is your ntop 4.0.3 configuration:
>
> Host System Type : x86_64-redhat-linux-gnu
> Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
> Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
> Defines : -DHAVE_CONFIG_H
> Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
> Include path :
> System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
> Locale : /usr/lib/locale
> Memory Debug : 3
>
> External packages:
>
> LBL pcap .h : standard system headers
> LBL pcap library : standard system libraries
> GNU gdbm .h : standard system headers
> GNU gdbm library : standard system libraries
> zlib .h : standard system headers
> zlib library : standard system libraries
> openSSL .h : standard system headers
> openSSL library : standard system libraries
>
> Install directories:
>
> Default prefix: /usr/local
> Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)
>
> Data files are in /usr/local/ntop-4.0.3/share/ntop
> Config files are in /usr/local/ntop-4.0.3/etc/ntop
> Run directory is /usr/local/ntop-4.0.3/var/ntop
> Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
> Database files are in /usr/local/ntop-4.0.3/var/ntop
>
> -------------------------------------------------------------------
>
> Here are some logs then that I was able to glean from the system log and/or gdb.
>
> 127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
> *** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
> aab41ede00 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x2abf39c0430f]
> /lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
> /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
> /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
> /lib64/libpthread.so.0[0x2abf3974473d]
> /lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]
>
> While running the main Ntop process in gdb:
>
> Detaching after fork from child process 22963.
> Detaching after fork from child process 22964.
> Detaching after fork from child process 22965.
> Detaching after fork from child process 22966.
> Detaching after fork from child process 22968.
> Detaching after fork from child process 22969.
> Detaching after fork from child process 22970.
> [...]
> Detaching after fork from child process 22971.
> *** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x2aaaab8b930f]
> /lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
> /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
> /lib64/libpthread.so.0[0x2aaaab3f973d]
> /lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
> ======= Memory map: ========
> 00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> 00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> 00611000-01892000 rw-p 00611000 00:00 0 [heap]
> 40000000-40001000 ---p 40000000 00:00 0
> 40001000-40a01000 rw-p 40001000 00:00 0
> 40a01000-40a02000 ---p 40a01000 00:00 0
> 40a02000-41402000 rw-p 40a02000 00:00 0
> 41402000-41403000 ---p 41402000 00:00 0
> 41403000-41e03000 rw-p 41403000 00:00 0
> 41e03000-41e04000 ---p 41e03000 00:00 0
> 41e04000-42804000 rw-p 41e04000 00:00 0
> 42804000-42805000 ---p 42804000 00:00 0
> 42805000-43205000 rw-p 42805000 00:00 0
> 43205000-43206000 ---p 43205000 00:00 0
> 43206000-43c06000 rw-p 43206000 00:00 0
> 43c06000-43c07000 ---p 43c06000 00:00 0
> [...]
> 2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
> 2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
> 2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
> 2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
> 2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
> 2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0x41e02940 (LWP 22660)]
> 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> (gdb)
> (gdb)
>
> (gdb) bt
> #0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> #1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
> #2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
> #3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
> #4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
> #5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
> #6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
> #7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
> (gdb)
>
> I am happy to provide any more information you might need or perform any
> additional debugging tasks that would help track this issue down.
>
> Thanks,
> Conor.
> --
> +---------------------------------------------------------------+
> It was a new day yesterday, but it's an old day now.
> MSN: schro5 [at] hotmail
> ICQ: 112562229
> GPG: http://www.konundrum.org/schro.asc
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop

--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


schrodinger at konundrum

Apr 12, 2011, 7:16 AM

Post #16 of 22 (2537 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

On 2011/04/12 08:44, Gary Gatten wrote:
> Certainly not an expert, but I see text like "dequeueAddress" and think name/ip lookups. Disable this (-z?) and see how things go. I've had issues with the resolution process that disabling fixed. Maybe some of those same issues aren't fully patched and are impacting you?

I would want the TCP session tracking though, this could be useful for
accounting and security. How would this affect Netflow accounting?

Conor.

>
> ----- Original Message -----
> From: Schrodinger [mailto:schrodinger [at] konundrum]
> Sent: Tuesday, April 12, 2011 08:23 AM
> To: ntop [at] listgateway <ntop [at] listgateway>
> Subject: Re: [Ntop] known double free bug in 4.0.3?
>
> On 2011/04/12 07:28, Gary Gatten wrote:
> > If it dies so often, perhaps try running it with a bunch of stuff disabled; name lookups, decoders, rrd (as much as possible), etc. and see if that makes a difference.
>
> Gary,
>
> Error messages below are from Ntop running normally, not in gdb and also
> running in gdb + gdb backtrace.
>
> "While running the main Ntop process in gdb"
>
> Conor.
>
> >
> > Have you tried running in gdb? I found many of my issues there, except once when it would die every hour running "normal", but running in gdb it would NOT crash! Typical :). The '-k (or -K?)' MAY have had something to do with that.
> >
> > ----- Original Message -----
> > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > Sent: Tuesday, April 12, 2011 04:01 AM
> > To: ntop [at] listgateway <ntop [at] listgateway>
> > Subject: Re: [Ntop] known double free bug in 4.0.3?
> >
> > Hi,
> >
> > I have this exact same problem, "double free or corruption". I have
> > compiled Ntop with debugging and electric fence and the problem still
> > persists. Usually Ntop can't last more than a few hours at a time :( I
> > am using Ntop to collect Netflow from different sources via thge native
> > Netflow plugin.
> >
> > I had compiled Ntop with additional debugging to try and locate the
> > problem.
> >
> > [user [at] hos ntop-4.0.3]# export CFLAGS+=-g
> > [user [at] hos ntop-4.0.3]# echo $CFLAGS
> > -g
> > [user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2
> >
> > ===================================================================
> >
> > This is your ntop 4.0.3 configuration:
> >
> > Host System Type : x86_64-redhat-linux-gnu
> > Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
> > Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
> > Defines : -DHAVE_CONFIG_H
> > Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
> > Include path :
> > System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
> > Locale : /usr/lib/locale
> > Memory Debug : 3
> >
> > External packages:
> >
> > LBL pcap .h : standard system headers
> > LBL pcap library : standard system libraries
> > GNU gdbm .h : standard system headers
> > GNU gdbm library : standard system libraries
> > zlib .h : standard system headers
> > zlib library : standard system libraries
> > openSSL .h : standard system headers
> > openSSL library : standard system libraries
> >
> > Install directories:
> >
> > Default prefix: /usr/local
> > Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)
> >
> > Data files are in /usr/local/ntop-4.0.3/share/ntop
> > Config files are in /usr/local/ntop-4.0.3/etc/ntop
> > Run directory is /usr/local/ntop-4.0.3/var/ntop
> > Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
> > Database files are in /usr/local/ntop-4.0.3/var/ntop
> >
> > -------------------------------------------------------------------
> >
> > Here are some logs then that I was able to glean from the system log and/or gdb.
> >
> > 127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
> > *** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
> > aab41ede00 ***
> > ======= Backtrace: =========
> > /lib64/libc.so.6[0x2abf39c0430f]
> > /lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
> > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
> > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
> > /lib64/libpthread.so.0[0x2abf3974473d]
> > /lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]
> >
> > While running the main Ntop process in gdb:
> >
> > Detaching after fork from child process 22963.
> > Detaching after fork from child process 22964.
> > Detaching after fork from child process 22965.
> > Detaching after fork from child process 22966.
> > Detaching after fork from child process 22968.
> > Detaching after fork from child process 22969.
> > Detaching after fork from child process 22970.
> > [...]
> > Detaching after fork from child process 22971.
> > *** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
> > ======= Backtrace: =========
> > /lib64/libc.so.6[0x2aaaab8b930f]
> > /lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
> > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
> > /lib64/libpthread.so.0[0x2aaaab3f973d]
> > /lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
> > ======= Memory map: ========
> > 00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > 00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > 00611000-01892000 rw-p 00611000 00:00 0 [heap]
> > 40000000-40001000 ---p 40000000 00:00 0
> > 40001000-40a01000 rw-p 40001000 00:00 0
> > 40a01000-40a02000 ---p 40a01000 00:00 0
> > 40a02000-41402000 rw-p 40a02000 00:00 0
> > 41402000-41403000 ---p 41402000 00:00 0
> > 41403000-41e03000 rw-p 41403000 00:00 0
> > 41e03000-41e04000 ---p 41e03000 00:00 0
> > 41e04000-42804000 rw-p 41e04000 00:00 0
> > 42804000-42805000 ---p 42804000 00:00 0
> > 42805000-43205000 rw-p 42805000 00:00 0
> > 43205000-43206000 ---p 43205000 00:00 0
> > 43206000-43c06000 rw-p 43206000 00:00 0
> > 43c06000-43c07000 ---p 43c06000 00:00 0
> > [...]
> > 2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
> > 2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
> > 2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
> > 2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
> > 2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
> > 2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
> > Program received signal SIGABRT, Aborted.
> > [Switching to Thread 0x41e02940 (LWP 22660)]
> > 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > (gdb)
> > (gdb)
> >
> > (gdb) bt
> > #0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > #1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
> > #2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
> > #3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
> > #4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
> > #5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
> > #6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
> > #7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
> > (gdb)
> >
> > I am happy to provide any more information you might need or perform any
> > additional debugging tasks that would help track this issue down.
> >
> > Thanks,
> > Conor.
> > --
> > +---------------------------------------------------------------+
> > It was a new day yesterday, but it's an old day now.
> > MSN: schro5 [at] hotmail
> > ICQ: 112562229
> > GPG: http://www.konundrum.org/schro.asc
> >
> >
> >
> >
> >
> > <font size="1">
> > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > </div>
> > "This email is intended to be reviewed by only the intended recipient
> > and may contain information that is privileged and/or confidential.
> > If you are not the intended recipient, you are hereby notified that
> > any review, use, dissemination, disclosure or copying of this email
> > and its attachments, if any, is strictly prohibited. If you have
> > received this email in error, please immediately notify the sender by
> > return email and delete this email from your system."
> > </font>
> >
> > _______________________________________________
> > Ntop mailing list
> > Ntop [at] listgateway
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> --
> +---------------------------------------------------------------+
> It was a new day yesterday, but it's an old day now.
> MSN: schro5 [at] hotmail
> ICQ: 112562229
> GPG: http://www.konundrum.org/schro.asc
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop

--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc


Ggatten at waddell

Apr 12, 2011, 7:51 AM

Post #17 of 22 (2539 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

Oh yeah, there's known issues with the tcp session tracking. At one point it was disabled in the code, is it back again?

So -z to disable sessions and -n to disable name/ip resolution. Start by disabling one, or both I guess. My bet is your problems go away.

When I was troubleshooting I had about a dozen different binaries with different features, such as disable IPv6, and each one of those a half dozen different startup args.

----- Original Message -----
From: Schrodinger [mailto:schrodinger [at] konundrum]
Sent: Tuesday, April 12, 2011 09:16 AM
To: ntop [at] listgateway <ntop [at] listgateway>
Subject: Re: [Ntop] known double free bug in 4.0.3?

On 2011/04/12 08:44, Gary Gatten wrote:
> Certainly not an expert, but I see text like "dequeueAddress" and think name/ip lookups. Disable this (-z?) and see how things go. I've had issues with the resolution process that disabling fixed. Maybe some of those same issues aren't fully patched and are impacting you?

I would want the TCP session tracking though, this could be useful for
accounting and security. How would this affect Netflow accounting?

Conor.

>
> ----- Original Message -----
> From: Schrodinger [mailto:schrodinger [at] konundrum]
> Sent: Tuesday, April 12, 2011 08:23 AM
> To: ntop [at] listgateway <ntop [at] listgateway>
> Subject: Re: [Ntop] known double free bug in 4.0.3?
>
> On 2011/04/12 07:28, Gary Gatten wrote:
> > If it dies so often, perhaps try running it with a bunch of stuff disabled; name lookups, decoders, rrd (as much as possible), etc. and see if that makes a difference.
>
> Gary,
>
> Error messages below are from Ntop running normally, not in gdb and also
> running in gdb + gdb backtrace.
>
> "While running the main Ntop process in gdb"
>
> Conor.
>
> >
> > Have you tried running in gdb? I found many of my issues there, except once when it would die every hour running "normal", but running in gdb it would NOT crash! Typical :). The '-k (or -K?)' MAY have had something to do with that.
> >
> > ----- Original Message -----
> > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > Sent: Tuesday, April 12, 2011 04:01 AM
> > To: ntop [at] listgateway <ntop [at] listgateway>
> > Subject: Re: [Ntop] known double free bug in 4.0.3?
> >
> > Hi,
> >
> > I have this exact same problem, "double free or corruption". I have
> > compiled Ntop with debugging and electric fence and the problem still
> > persists. Usually Ntop can't last more than a few hours at a time :( I
> > am using Ntop to collect Netflow from different sources via thge native
> > Netflow plugin.
> >
> > I had compiled Ntop with additional debugging to try and locate the
> > problem.
> >
> > [user [at] hos ntop-4.0.3]# export CFLAGS+=-g
> > [user [at] hos ntop-4.0.3]# echo $CFLAGS
> > -g
> > [user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2
> >
> > ===================================================================
> >
> > This is your ntop 4.0.3 configuration:
> >
> > Host System Type : x86_64-redhat-linux-gnu
> > Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
> > Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
> > Defines : -DHAVE_CONFIG_H
> > Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
> > Include path :
> > System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
> > Locale : /usr/lib/locale
> > Memory Debug : 3
> >
> > External packages:
> >
> > LBL pcap .h : standard system headers
> > LBL pcap library : standard system libraries
> > GNU gdbm .h : standard system headers
> > GNU gdbm library : standard system libraries
> > zlib .h : standard system headers
> > zlib library : standard system libraries
> > openSSL .h : standard system headers
> > openSSL library : standard system libraries
> >
> > Install directories:
> >
> > Default prefix: /usr/local
> > Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)
> >
> > Data files are in /usr/local/ntop-4.0.3/share/ntop
> > Config files are in /usr/local/ntop-4.0.3/etc/ntop
> > Run directory is /usr/local/ntop-4.0.3/var/ntop
> > Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
> > Database files are in /usr/local/ntop-4.0.3/var/ntop
> >
> > -------------------------------------------------------------------
> >
> > Here are some logs then that I was able to glean from the system log and/or gdb.
> >
> > 127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
> > *** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
> > aab41ede00 ***
> > ======= Backtrace: =========
> > /lib64/libc.so.6[0x2abf39c0430f]
> > /lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
> > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
> > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
> > /lib64/libpthread.so.0[0x2abf3974473d]
> > /lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]
> >
> > While running the main Ntop process in gdb:
> >
> > Detaching after fork from child process 22963.
> > Detaching after fork from child process 22964.
> > Detaching after fork from child process 22965.
> > Detaching after fork from child process 22966.
> > Detaching after fork from child process 22968.
> > Detaching after fork from child process 22969.
> > Detaching after fork from child process 22970.
> > [...]
> > Detaching after fork from child process 22971.
> > *** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
> > ======= Backtrace: =========
> > /lib64/libc.so.6[0x2aaaab8b930f]
> > /lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
> > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
> > /lib64/libpthread.so.0[0x2aaaab3f973d]
> > /lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
> > ======= Memory map: ========
> > 00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > 00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > 00611000-01892000 rw-p 00611000 00:00 0 [heap]
> > 40000000-40001000 ---p 40000000 00:00 0
> > 40001000-40a01000 rw-p 40001000 00:00 0
> > 40a01000-40a02000 ---p 40a01000 00:00 0
> > 40a02000-41402000 rw-p 40a02000 00:00 0
> > 41402000-41403000 ---p 41402000 00:00 0
> > 41403000-41e03000 rw-p 41403000 00:00 0
> > 41e03000-41e04000 ---p 41e03000 00:00 0
> > 41e04000-42804000 rw-p 41e04000 00:00 0
> > 42804000-42805000 ---p 42804000 00:00 0
> > 42805000-43205000 rw-p 42805000 00:00 0
> > 43205000-43206000 ---p 43205000 00:00 0
> > 43206000-43c06000 rw-p 43206000 00:00 0
> > 43c06000-43c07000 ---p 43c06000 00:00 0
> > [...]
> > 2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
> > 2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
> > 2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
> > 2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
> > 2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
> > 2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
> > Program received signal SIGABRT, Aborted.
> > [Switching to Thread 0x41e02940 (LWP 22660)]
> > 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > (gdb)
> > (gdb)
> >
> > (gdb) bt
> > #0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > #1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
> > #2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
> > #3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
> > #4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
> > #5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
> > #6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
> > #7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
> > (gdb)
> >
> > I am happy to provide any more information you might need or perform any
> > additional debugging tasks that would help track this issue down.
> >
> > Thanks,
> > Conor.
> > --
> > +---------------------------------------------------------------+
> > It was a new day yesterday, but it's an old day now.
> > MSN: schro5 [at] hotmail
> > ICQ: 112562229
> > GPG: http://www.konundrum.org/schro.asc
> >
> >
> >
> >
> >
> > <font size="1">
> > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > </div>
> > "This email is intended to be reviewed by only the intended recipient
> > and may contain information that is privileged and/or confidential.
> > If you are not the intended recipient, you are hereby notified that
> > any review, use, dissemination, disclosure or copying of this email
> > and its attachments, if any, is strictly prohibited. If you have
> > received this email in error, please immediately notify the sender by
> > return email and delete this email from your system."
> > </font>
> >
> > _______________________________________________
> > Ntop mailing list
> > Ntop [at] listgateway
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> --
> +---------------------------------------------------------------+
> It was a new day yesterday, but it's an old day now.
> MSN: schro5 [at] hotmail
> ICQ: 112562229
> GPG: http://www.konundrum.org/schro.asc
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop

--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


schrodinger at konundrum

Apr 13, 2011, 2:44 AM

Post #18 of 22 (2535 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

On 2011/04/12 09:51, Gary Gatten wrote:
> Oh yeah, there's known issues with the tcp session tracking. At one point it was disabled in the code, is it back again?
>
> So -z to disable sessions and -n to disable name/ip resolution. Start by disabling one, or both I guess. My bet is your problems go away.

I ran Ntop yesterday with -z and came in this morning to find it has
crashed again.

I am running with -n 0 and -z now to see how it goes.

Conor.

>
> When I was troubleshooting I had about a dozen different binaries with different features, such as disable IPv6, and each one of those a half dozen different startup args.
>
> ----- Original Message -----
> From: Schrodinger [mailto:schrodinger [at] konundrum]
> Sent: Tuesday, April 12, 2011 09:16 AM
> To: ntop [at] listgateway <ntop [at] listgateway>
> Subject: Re: [Ntop] known double free bug in 4.0.3?
>
> On 2011/04/12 08:44, Gary Gatten wrote:
> > Certainly not an expert, but I see text like "dequeueAddress" and think name/ip lookups. Disable this (-z?) and see how things go. I've had issues with the resolution process that disabling fixed. Maybe some of those same issues aren't fully patched and are impacting you?
>
> I would want the TCP session tracking though, this could be useful for
> accounting and security. How would this affect Netflow accounting?
>
> Conor.
>
> >
> > ----- Original Message -----
> > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > Sent: Tuesday, April 12, 2011 08:23 AM
> > To: ntop [at] listgateway <ntop [at] listgateway>
> > Subject: Re: [Ntop] known double free bug in 4.0.3?
> >
> > On 2011/04/12 07:28, Gary Gatten wrote:
> > > If it dies so often, perhaps try running it with a bunch of stuff disabled; name lookups, decoders, rrd (as much as possible), etc. and see if that makes a difference.
> >
> > Gary,
> >
> > Error messages below are from Ntop running normally, not in gdb and also
> > running in gdb + gdb backtrace.
> >
> > "While running the main Ntop process in gdb"
> >
> > Conor.
> >
> > >
> > > Have you tried running in gdb? I found many of my issues there, except once when it would die every hour running "normal", but running in gdb it would NOT crash! Typical :). The '-k (or -K?)' MAY have had something to do with that.
> > >
> > > ----- Original Message -----
> > > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > > Sent: Tuesday, April 12, 2011 04:01 AM
> > > To: ntop [at] listgateway <ntop [at] listgateway>
> > > Subject: Re: [Ntop] known double free bug in 4.0.3?
> > >
> > > Hi,
> > >
> > > I have this exact same problem, "double free or corruption". I have
> > > compiled Ntop with debugging and electric fence and the problem still
> > > persists. Usually Ntop can't last more than a few hours at a time :( I
> > > am using Ntop to collect Netflow from different sources via thge native
> > > Netflow plugin.
> > >
> > > I had compiled Ntop with additional debugging to try and locate the
> > > problem.
> > >
> > > [user [at] hos ntop-4.0.3]# export CFLAGS+=-g
> > > [user [at] hos ntop-4.0.3]# echo $CFLAGS
> > > -g
> > > [user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2
> > >
> > > ===================================================================
> > >
> > > This is your ntop 4.0.3 configuration:
> > >
> > > Host System Type : x86_64-redhat-linux-gnu
> > > Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
> > > Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
> > > Defines : -DHAVE_CONFIG_H
> > > Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
> > > Include path :
> > > System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
> > > Locale : /usr/lib/locale
> > > Memory Debug : 3
> > >
> > > External packages:
> > >
> > > LBL pcap .h : standard system headers
> > > LBL pcap library : standard system libraries
> > > GNU gdbm .h : standard system headers
> > > GNU gdbm library : standard system libraries
> > > zlib .h : standard system headers
> > > zlib library : standard system libraries
> > > openSSL .h : standard system headers
> > > openSSL library : standard system libraries
> > >
> > > Install directories:
> > >
> > > Default prefix: /usr/local
> > > Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)
> > >
> > > Data files are in /usr/local/ntop-4.0.3/share/ntop
> > > Config files are in /usr/local/ntop-4.0.3/etc/ntop
> > > Run directory is /usr/local/ntop-4.0.3/var/ntop
> > > Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
> > > Database files are in /usr/local/ntop-4.0.3/var/ntop
> > >
> > > -------------------------------------------------------------------
> > >
> > > Here are some logs then that I was able to glean from the system log and/or gdb.
> > >
> > > 127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
> > > *** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
> > > aab41ede00 ***
> > > ======= Backtrace: =========
> > > /lib64/libc.so.6[0x2abf39c0430f]
> > > /lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
> > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
> > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
> > > /lib64/libpthread.so.0[0x2abf3974473d]
> > > /lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]
> > >
> > > While running the main Ntop process in gdb:
> > >
> > > Detaching after fork from child process 22963.
> > > Detaching after fork from child process 22964.
> > > Detaching after fork from child process 22965.
> > > Detaching after fork from child process 22966.
> > > Detaching after fork from child process 22968.
> > > Detaching after fork from child process 22969.
> > > Detaching after fork from child process 22970.
> > > [...]
> > > Detaching after fork from child process 22971.
> > > *** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
> > > ======= Backtrace: =========
> > > /lib64/libc.so.6[0x2aaaab8b930f]
> > > /lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
> > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
> > > /lib64/libpthread.so.0[0x2aaaab3f973d]
> > > /lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
> > > ======= Memory map: ========
> > > 00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > > 00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > > 00611000-01892000 rw-p 00611000 00:00 0 [heap]
> > > 40000000-40001000 ---p 40000000 00:00 0
> > > 40001000-40a01000 rw-p 40001000 00:00 0
> > > 40a01000-40a02000 ---p 40a01000 00:00 0
> > > 40a02000-41402000 rw-p 40a02000 00:00 0
> > > 41402000-41403000 ---p 41402000 00:00 0
> > > 41403000-41e03000 rw-p 41403000 00:00 0
> > > 41e03000-41e04000 ---p 41e03000 00:00 0
> > > 41e04000-42804000 rw-p 41e04000 00:00 0
> > > 42804000-42805000 ---p 42804000 00:00 0
> > > 42805000-43205000 rw-p 42805000 00:00 0
> > > 43205000-43206000 ---p 43205000 00:00 0
> > > 43206000-43c06000 rw-p 43206000 00:00 0
> > > 43c06000-43c07000 ---p 43c06000 00:00 0
> > > [...]
> > > 2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
> > > 2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
> > > 2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
> > > 2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
> > > 2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
> > > 2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
> > > Program received signal SIGABRT, Aborted.
> > > [Switching to Thread 0x41e02940 (LWP 22660)]
> > > 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > > (gdb)
> > > (gdb)
> > >
> > > (gdb) bt
> > > #0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > > #1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
> > > #2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
> > > #3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
> > > #4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
> > > #5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
> > > #6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
> > > #7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
> > > (gdb)
> > >
> > > I am happy to provide any more information you might need or perform any
> > > additional debugging tasks that would help track this issue down.
> > >
> > > Thanks,
> > > Conor.
> > > --
> > > +---------------------------------------------------------------+
> > > It was a new day yesterday, but it's an old day now.
> > > MSN: schro5 [at] hotmail
> > > ICQ: 112562229
> > > GPG: http://www.konundrum.org/schro.asc
> > >
> > >
> > >
> > >
> > >
> > > <font size="1">
> > > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > > </div>
> > > "This email is intended to be reviewed by only the intended recipient
> > > and may contain information that is privileged and/or confidential.
> > > If you are not the intended recipient, you are hereby notified that
> > > any review, use, dissemination, disclosure or copying of this email
> > > and its attachments, if any, is strictly prohibited. If you have
> > > received this email in error, please immediately notify the sender by
> > > return email and delete this email from your system."
> > > </font>
> > >
> > > _______________________________________________
> > > Ntop mailing list
> > > Ntop [at] listgateway
> > > http://listgateway.unipi.it/mailman/listinfo/ntop
> >
> > --
> > +---------------------------------------------------------------+
> > It was a new day yesterday, but it's an old day now.
> > MSN: schro5 [at] hotmail
> > ICQ: 112562229
> > GPG: http://www.konundrum.org/schro.asc
> >
> >
> >
> >
> >
> > <font size="1">
> > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > </div>
> > "This email is intended to be reviewed by only the intended recipient
> > and may contain information that is privileged and/or confidential.
> > If you are not the intended recipient, you are hereby notified that
> > any review, use, dissemination, disclosure or copying of this email
> > and its attachments, if any, is strictly prohibited. If you have
> > received this email in error, please immediately notify the sender by
> > return email and delete this email from your system."
> > </font>
> >
> > _______________________________________________
> > Ntop mailing list
> > Ntop [at] listgateway
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> --
> +---------------------------------------------------------------+
> It was a new day yesterday, but it's an old day now.
> MSN: schro5 [at] hotmail
> ICQ: 112562229
> GPG: http://www.konundrum.org/schro.asc
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop

--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc


Ggatten at waddell

Apr 13, 2011, 3:28 AM

Post #19 of 22 (2527 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

What's "-n 0" ? I don't have 4.0.3, but I just have "-n". I'm curious to see what you find. When I have ip lookups / name res enabled it's not good, consumes "all" cpu and eventually dies. I'm several revs behind though so it will be interesting to see how this impacts you.



----- Original Message -----
From: Schrodinger [mailto:schrodinger [at] konundrum]
Sent: Wednesday, April 13, 2011 04:44 AM
To: ntop [at] listgateway <ntop [at] listgateway>
Subject: Re: [Ntop] known double free bug in 4.0.3?

On 2011/04/12 09:51, Gary Gatten wrote:
> Oh yeah, there's known issues with the tcp session tracking. At one point it was disabled in the code, is it back again?
>
> So -z to disable sessions and -n to disable name/ip resolution. Start by disabling one, or both I guess. My bet is your problems go away.

I ran Ntop yesterday with -z and came in this morning to find it has
crashed again.

I am running with -n 0 and -z now to see how it goes.

Conor.

>
> When I was troubleshooting I had about a dozen different binaries with different features, such as disable IPv6, and each one of those a half dozen different startup args.
>
> ----- Original Message -----
> From: Schrodinger [mailto:schrodinger [at] konundrum]
> Sent: Tuesday, April 12, 2011 09:16 AM
> To: ntop [at] listgateway <ntop [at] listgateway>
> Subject: Re: [Ntop] known double free bug in 4.0.3?
>
> On 2011/04/12 08:44, Gary Gatten wrote:
> > Certainly not an expert, but I see text like "dequeueAddress" and think name/ip lookups. Disable this (-z?) and see how things go. I've had issues with the resolution process that disabling fixed. Maybe some of those same issues aren't fully patched and are impacting you?
>
> I would want the TCP session tracking though, this could be useful for
> accounting and security. How would this affect Netflow accounting?
>
> Conor.
>
> >
> > ----- Original Message -----
> > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > Sent: Tuesday, April 12, 2011 08:23 AM
> > To: ntop [at] listgateway <ntop [at] listgateway>
> > Subject: Re: [Ntop] known double free bug in 4.0.3?
> >
> > On 2011/04/12 07:28, Gary Gatten wrote:
> > > If it dies so often, perhaps try running it with a bunch of stuff disabled; name lookups, decoders, rrd (as much as possible), etc. and see if that makes a difference.
> >
> > Gary,
> >
> > Error messages below are from Ntop running normally, not in gdb and also
> > running in gdb + gdb backtrace.
> >
> > "While running the main Ntop process in gdb"
> >
> > Conor.
> >
> > >
> > > Have you tried running in gdb? I found many of my issues there, except once when it would die every hour running "normal", but running in gdb it would NOT crash! Typical :). The '-k (or -K?)' MAY have had something to do with that.
> > >
> > > ----- Original Message -----
> > > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > > Sent: Tuesday, April 12, 2011 04:01 AM
> > > To: ntop [at] listgateway <ntop [at] listgateway>
> > > Subject: Re: [Ntop] known double free bug in 4.0.3?
> > >
> > > Hi,
> > >
> > > I have this exact same problem, "double free or corruption". I have
> > > compiled Ntop with debugging and electric fence and the problem still
> > > persists. Usually Ntop can't last more than a few hours at a time :( I
> > > am using Ntop to collect Netflow from different sources via thge native
> > > Netflow plugin.
> > >
> > > I had compiled Ntop with additional debugging to try and locate the
> > > problem.
> > >
> > > [user [at] hos ntop-4.0.3]# export CFLAGS+=-g
> > > [user [at] hos ntop-4.0.3]# echo $CFLAGS
> > > -g
> > > [user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2
> > >
> > > ===================================================================
> > >
> > > This is your ntop 4.0.3 configuration:
> > >
> > > Host System Type : x86_64-redhat-linux-gnu
> > > Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
> > > Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
> > > Defines : -DHAVE_CONFIG_H
> > > Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
> > > Include path :
> > > System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
> > > Locale : /usr/lib/locale
> > > Memory Debug : 3
> > >
> > > External packages:
> > >
> > > LBL pcap .h : standard system headers
> > > LBL pcap library : standard system libraries
> > > GNU gdbm .h : standard system headers
> > > GNU gdbm library : standard system libraries
> > > zlib .h : standard system headers
> > > zlib library : standard system libraries
> > > openSSL .h : standard system headers
> > > openSSL library : standard system libraries
> > >
> > > Install directories:
> > >
> > > Default prefix: /usr/local
> > > Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)
> > >
> > > Data files are in /usr/local/ntop-4.0.3/share/ntop
> > > Config files are in /usr/local/ntop-4.0.3/etc/ntop
> > > Run directory is /usr/local/ntop-4.0.3/var/ntop
> > > Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
> > > Database files are in /usr/local/ntop-4.0.3/var/ntop
> > >
> > > -------------------------------------------------------------------
> > >
> > > Here are some logs then that I was able to glean from the system log and/or gdb.
> > >
> > > 127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
> > > *** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
> > > aab41ede00 ***
> > > ======= Backtrace: =========
> > > /lib64/libc.so.6[0x2abf39c0430f]
> > > /lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
> > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
> > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
> > > /lib64/libpthread.so.0[0x2abf3974473d]
> > > /lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]
> > >
> > > While running the main Ntop process in gdb:
> > >
> > > Detaching after fork from child process 22963.
> > > Detaching after fork from child process 22964.
> > > Detaching after fork from child process 22965.
> > > Detaching after fork from child process 22966.
> > > Detaching after fork from child process 22968.
> > > Detaching after fork from child process 22969.
> > > Detaching after fork from child process 22970.
> > > [...]
> > > Detaching after fork from child process 22971.
> > > *** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
> > > ======= Backtrace: =========
> > > /lib64/libc.so.6[0x2aaaab8b930f]
> > > /lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
> > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
> > > /lib64/libpthread.so.0[0x2aaaab3f973d]
> > > /lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
> > > ======= Memory map: ========
> > > 00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > > 00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > > 00611000-01892000 rw-p 00611000 00:00 0 [heap]
> > > 40000000-40001000 ---p 40000000 00:00 0
> > > 40001000-40a01000 rw-p 40001000 00:00 0
> > > 40a01000-40a02000 ---p 40a01000 00:00 0
> > > 40a02000-41402000 rw-p 40a02000 00:00 0
> > > 41402000-41403000 ---p 41402000 00:00 0
> > > 41403000-41e03000 rw-p 41403000 00:00 0
> > > 41e03000-41e04000 ---p 41e03000 00:00 0
> > > 41e04000-42804000 rw-p 41e04000 00:00 0
> > > 42804000-42805000 ---p 42804000 00:00 0
> > > 42805000-43205000 rw-p 42805000 00:00 0
> > > 43205000-43206000 ---p 43205000 00:00 0
> > > 43206000-43c06000 rw-p 43206000 00:00 0
> > > 43c06000-43c07000 ---p 43c06000 00:00 0
> > > [...]
> > > 2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
> > > 2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
> > > 2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
> > > 2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
> > > 2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
> > > 2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
> > > Program received signal SIGABRT, Aborted.
> > > [Switching to Thread 0x41e02940 (LWP 22660)]
> > > 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > > (gdb)
> > > (gdb)
> > >
> > > (gdb) bt
> > > #0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > > #1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
> > > #2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
> > > #3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
> > > #4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
> > > #5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
> > > #6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
> > > #7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
> > > (gdb)
> > >
> > > I am happy to provide any more information you might need or perform any
> > > additional debugging tasks that would help track this issue down.
> > >
> > > Thanks,
> > > Conor.
> > > --
> > > +---------------------------------------------------------------+
> > > It was a new day yesterday, but it's an old day now.
> > > MSN: schro5 [at] hotmail
> > > ICQ: 112562229
> > > GPG: http://www.konundrum.org/schro.asc
> > >
> > >
> > >
> > >
> > >
> > > <font size="1">
> > > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > > </div>
> > > "This email is intended to be reviewed by only the intended recipient
> > > and may contain information that is privileged and/or confidential.
> > > If you are not the intended recipient, you are hereby notified that
> > > any review, use, dissemination, disclosure or copying of this email
> > > and its attachments, if any, is strictly prohibited. If you have
> > > received this email in error, please immediately notify the sender by
> > > return email and delete this email from your system."
> > > </font>
> > >
> > > _______________________________________________
> > > Ntop mailing list
> > > Ntop [at] listgateway
> > > http://listgateway.unipi.it/mailman/listinfo/ntop
> >
> > --
> > +---------------------------------------------------------------+
> > It was a new day yesterday, but it's an old day now.
> > MSN: schro5 [at] hotmail
> > ICQ: 112562229
> > GPG: http://www.konundrum.org/schro.asc
> >
> >
> >
> >
> >
> > <font size="1">
> > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > </div>
> > "This email is intended to be reviewed by only the intended recipient
> > and may contain information that is privileged and/or confidential.
> > If you are not the intended recipient, you are hereby notified that
> > any review, use, dissemination, disclosure or copying of this email
> > and its attachments, if any, is strictly prohibited. If you have
> > received this email in error, please immediately notify the sender by
> > return email and delete this email from your system."
> > </font>
> >
> > _______________________________________________
> > Ntop mailing list
> > Ntop [at] listgateway
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> --
> +---------------------------------------------------------------+
> It was a new day yesterday, but it's an old day now.
> MSN: schro5 [at] hotmail
> ICQ: 112562229
> GPG: http://www.konundrum.org/schro.asc
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop

--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


schrodinger at konundrum

Apr 13, 2011, 4:45 AM

Post #20 of 22 (2536 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

On 2011/04/13 05:28, Gary Gatten wrote:
> What's "-n 0" ? I don't have 4.0.3, but I just have "-n". I'm curious to see what you find. When I have ip lookups / name res enabled it's not good, consumes "all" cpu and eventually dies. I'm several revs behind though so it will be interesting to see how this impacts you.

[-n <mode> | --numeric-ip-addresses <mode>]
Numeric IP addresses DNS resolution mode:
0 - No DNS resolution at all
1 - DNS resolution for local hosts only
2 - DNS resolution for remote hosts only

Ntop still running so far.

Conor.

>
>
>
> ----- Original Message -----
> From: Schrodinger [mailto:schrodinger [at] konundrum]
> Sent: Wednesday, April 13, 2011 04:44 AM
> To: ntop [at] listgateway <ntop [at] listgateway>
> Subject: Re: [Ntop] known double free bug in 4.0.3?
>
> On 2011/04/12 09:51, Gary Gatten wrote:
> > Oh yeah, there's known issues with the tcp session tracking. At one point it was disabled in the code, is it back again?
> >
> > So -z to disable sessions and -n to disable name/ip resolution. Start by disabling one, or both I guess. My bet is your problems go away.
>
> I ran Ntop yesterday with -z and came in this morning to find it has
> crashed again.
>
> I am running with -n 0 and -z now to see how it goes.
>
> Conor.
>
> >
> > When I was troubleshooting I had about a dozen different binaries with different features, such as disable IPv6, and each one of those a half dozen different startup args.
> >
> > ----- Original Message -----
> > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > Sent: Tuesday, April 12, 2011 09:16 AM
> > To: ntop [at] listgateway <ntop [at] listgateway>
> > Subject: Re: [Ntop] known double free bug in 4.0.3?
> >
> > On 2011/04/12 08:44, Gary Gatten wrote:
> > > Certainly not an expert, but I see text like "dequeueAddress" and think name/ip lookups. Disable this (-z?) and see how things go. I've had issues with the resolution process that disabling fixed. Maybe some of those same issues aren't fully patched and are impacting you?
> >
> > I would want the TCP session tracking though, this could be useful for
> > accounting and security. How would this affect Netflow accounting?
> >
> > Conor.
> >
> > >
> > > ----- Original Message -----
> > > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > > Sent: Tuesday, April 12, 2011 08:23 AM
> > > To: ntop [at] listgateway <ntop [at] listgateway>
> > > Subject: Re: [Ntop] known double free bug in 4.0.3?
> > >
> > > On 2011/04/12 07:28, Gary Gatten wrote:
> > > > If it dies so often, perhaps try running it with a bunch of stuff disabled; name lookups, decoders, rrd (as much as possible), etc. and see if that makes a difference.
> > >
> > > Gary,
> > >
> > > Error messages below are from Ntop running normally, not in gdb and also
> > > running in gdb + gdb backtrace.
> > >
> > > "While running the main Ntop process in gdb"
> > >
> > > Conor.
> > >
> > > >
> > > > Have you tried running in gdb? I found many of my issues there, except once when it would die every hour running "normal", but running in gdb it would NOT crash! Typical :). The '-k (or -K?)' MAY have had something to do with that.
> > > >
> > > > ----- Original Message -----
> > > > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > > > Sent: Tuesday, April 12, 2011 04:01 AM
> > > > To: ntop [at] listgateway <ntop [at] listgateway>
> > > > Subject: Re: [Ntop] known double free bug in 4.0.3?
> > > >
> > > > Hi,
> > > >
> > > > I have this exact same problem, "double free or corruption". I have
> > > > compiled Ntop with debugging and electric fence and the problem still
> > > > persists. Usually Ntop can't last more than a few hours at a time :( I
> > > > am using Ntop to collect Netflow from different sources via thge native
> > > > Netflow plugin.
> > > >
> > > > I had compiled Ntop with additional debugging to try and locate the
> > > > problem.
> > > >
> > > > [user [at] hos ntop-4.0.3]# export CFLAGS+=-g
> > > > [user [at] hos ntop-4.0.3]# echo $CFLAGS
> > > > -g
> > > > [user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2
> > > >
> > > > ===================================================================
> > > >
> > > > This is your ntop 4.0.3 configuration:
> > > >
> > > > Host System Type : x86_64-redhat-linux-gnu
> > > > Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
> > > > Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
> > > > Defines : -DHAVE_CONFIG_H
> > > > Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
> > > > Include path :
> > > > System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
> > > > Locale : /usr/lib/locale
> > > > Memory Debug : 3
> > > >
> > > > External packages:
> > > >
> > > > LBL pcap .h : standard system headers
> > > > LBL pcap library : standard system libraries
> > > > GNU gdbm .h : standard system headers
> > > > GNU gdbm library : standard system libraries
> > > > zlib .h : standard system headers
> > > > zlib library : standard system libraries
> > > > openSSL .h : standard system headers
> > > > openSSL library : standard system libraries
> > > >
> > > > Install directories:
> > > >
> > > > Default prefix: /usr/local
> > > > Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)
> > > >
> > > > Data files are in /usr/local/ntop-4.0.3/share/ntop
> > > > Config files are in /usr/local/ntop-4.0.3/etc/ntop
> > > > Run directory is /usr/local/ntop-4.0.3/var/ntop
> > > > Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
> > > > Database files are in /usr/local/ntop-4.0.3/var/ntop
> > > >
> > > > -------------------------------------------------------------------
> > > >
> > > > Here are some logs then that I was able to glean from the system log and/or gdb.
> > > >
> > > > 127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
> > > > *** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
> > > > aab41ede00 ***
> > > > ======= Backtrace: =========
> > > > /lib64/libc.so.6[0x2abf39c0430f]
> > > > /lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
> > > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
> > > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
> > > > /lib64/libpthread.so.0[0x2abf3974473d]
> > > > /lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]
> > > >
> > > > While running the main Ntop process in gdb:
> > > >
> > > > Detaching after fork from child process 22963.
> > > > Detaching after fork from child process 22964.
> > > > Detaching after fork from child process 22965.
> > > > Detaching after fork from child process 22966.
> > > > Detaching after fork from child process 22968.
> > > > Detaching after fork from child process 22969.
> > > > Detaching after fork from child process 22970.
> > > > [...]
> > > > Detaching after fork from child process 22971.
> > > > *** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
> > > > ======= Backtrace: =========
> > > > /lib64/libc.so.6[0x2aaaab8b930f]
> > > > /lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
> > > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
> > > > /lib64/libpthread.so.0[0x2aaaab3f973d]
> > > > /lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
> > > > ======= Memory map: ========
> > > > 00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > > > 00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > > > 00611000-01892000 rw-p 00611000 00:00 0 [heap]
> > > > 40000000-40001000 ---p 40000000 00:00 0
> > > > 40001000-40a01000 rw-p 40001000 00:00 0
> > > > 40a01000-40a02000 ---p 40a01000 00:00 0
> > > > 40a02000-41402000 rw-p 40a02000 00:00 0
> > > > 41402000-41403000 ---p 41402000 00:00 0
> > > > 41403000-41e03000 rw-p 41403000 00:00 0
> > > > 41e03000-41e04000 ---p 41e03000 00:00 0
> > > > 41e04000-42804000 rw-p 41e04000 00:00 0
> > > > 42804000-42805000 ---p 42804000 00:00 0
> > > > 42805000-43205000 rw-p 42805000 00:00 0
> > > > 43205000-43206000 ---p 43205000 00:00 0
> > > > 43206000-43c06000 rw-p 43206000 00:00 0
> > > > 43c06000-43c07000 ---p 43c06000 00:00 0
> > > > [...]
> > > > 2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
> > > > 2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
> > > > 2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
> > > > 2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
> > > > 2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
> > > > 2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
> > > > Program received signal SIGABRT, Aborted.
> > > > [Switching to Thread 0x41e02940 (LWP 22660)]
> > > > 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > > > (gdb)
> > > > (gdb)
> > > >
> > > > (gdb) bt
> > > > #0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > > > #1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
> > > > #2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
> > > > #3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
> > > > #4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
> > > > #5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
> > > > #6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
> > > > #7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
> > > > (gdb)
> > > >
> > > > I am happy to provide any more information you might need or perform any
> > > > additional debugging tasks that would help track this issue down.
> > > >
> > > > Thanks,
> > > > Conor.
> > > > --
> > > > +---------------------------------------------------------------+
> > > > It was a new day yesterday, but it's an old day now.
> > > > MSN: schro5 [at] hotmail
> > > > ICQ: 112562229
> > > > GPG: http://www.konundrum.org/schro.asc
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > <font size="1">
> > > > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > > > </div>
> > > > "This email is intended to be reviewed by only the intended recipient
> > > > and may contain information that is privileged and/or confidential.
> > > > If you are not the intended recipient, you are hereby notified that
> > > > any review, use, dissemination, disclosure or copying of this email
> > > > and its attachments, if any, is strictly prohibited. If you have
> > > > received this email in error, please immediately notify the sender by
> > > > return email and delete this email from your system."
> > > > </font>
> > > >
> > > > _______________________________________________
> > > > Ntop mailing list
> > > > Ntop [at] listgateway
> > > > http://listgateway.unipi.it/mailman/listinfo/ntop
> > >
> > > --
> > > +---------------------------------------------------------------+
> > > It was a new day yesterday, but it's an old day now.
> > > MSN: schro5 [at] hotmail
> > > ICQ: 112562229
> > > GPG: http://www.konundrum.org/schro.asc
> > >
> > >
> > >
> > >
> > >
> > > <font size="1">
> > > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > > </div>
> > > "This email is intended to be reviewed by only the intended recipient
> > > and may contain information that is privileged and/or confidential.
> > > If you are not the intended recipient, you are hereby notified that
> > > any review, use, dissemination, disclosure or copying of this email
> > > and its attachments, if any, is strictly prohibited. If you have
> > > received this email in error, please immediately notify the sender by
> > > return email and delete this email from your system."
> > > </font>
> > >
> > > _______________________________________________
> > > Ntop mailing list
> > > Ntop [at] listgateway
> > > http://listgateway.unipi.it/mailman/listinfo/ntop
> >
> > --
> > +---------------------------------------------------------------+
> > It was a new day yesterday, but it's an old day now.
> > MSN: schro5 [at] hotmail
> > ICQ: 112562229
> > GPG: http://www.konundrum.org/schro.asc
> >
> >
> >
> >
> >
> > <font size="1">
> > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > </div>
> > "This email is intended to be reviewed by only the intended recipient
> > and may contain information that is privileged and/or confidential.
> > If you are not the intended recipient, you are hereby notified that
> > any review, use, dissemination, disclosure or copying of this email
> > and its attachments, if any, is strictly prohibited. If you have
> > received this email in error, please immediately notify the sender by
> > return email and delete this email from your system."
> > </font>
> >
> > _______________________________________________
> > Ntop mailing list
> > Ntop [at] listgateway
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> --
> +---------------------------------------------------------------+
> It was a new day yesterday, but it's an old day now.
> MSN: schro5 [at] hotmail
> ICQ: 112562229
> GPG: http://www.konundrum.org/schro.asc
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop

--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc


Ggatten at waddell

Apr 13, 2011, 6:11 AM

Post #21 of 22 (2530 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

Sweet! I recall asking for similar functionality some time ago. I'll have to drop a note to Luca. Looks like I need to upgrade and play with the new version a bit.

Thx! Gary


----- Original Message -----
From: Schrodinger [mailto:schrodinger [at] konundrum]
Sent: Wednesday, April 13, 2011 06:45 AM
To: ntop [at] listgateway <ntop [at] listgateway>
Subject: Re: [Ntop] known double free bug in 4.0.3?

On 2011/04/13 05:28, Gary Gatten wrote:
> What's "-n 0" ? I don't have 4.0.3, but I just have "-n". I'm curious to see what you find. When I have ip lookups / name res enabled it's not good, consumes "all" cpu and eventually dies. I'm several revs behind though so it will be interesting to see how this impacts you.

[-n <mode> | --numeric-ip-addresses <mode>]
Numeric IP addresses DNS resolution mode:
0 - No DNS resolution at all
1 - DNS resolution for local hosts only
2 - DNS resolution for remote hosts only

Ntop still running so far.

Conor.

>
>
>
> ----- Original Message -----
> From: Schrodinger [mailto:schrodinger [at] konundrum]
> Sent: Wednesday, April 13, 2011 04:44 AM
> To: ntop [at] listgateway <ntop [at] listgateway>
> Subject: Re: [Ntop] known double free bug in 4.0.3?
>
> On 2011/04/12 09:51, Gary Gatten wrote:
> > Oh yeah, there's known issues with the tcp session tracking. At one point it was disabled in the code, is it back again?
> >
> > So -z to disable sessions and -n to disable name/ip resolution. Start by disabling one, or both I guess. My bet is your problems go away.
>
> I ran Ntop yesterday with -z and came in this morning to find it has
> crashed again.
>
> I am running with -n 0 and -z now to see how it goes.
>
> Conor.
>
> >
> > When I was troubleshooting I had about a dozen different binaries with different features, such as disable IPv6, and each one of those a half dozen different startup args.
> >
> > ----- Original Message -----
> > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > Sent: Tuesday, April 12, 2011 09:16 AM
> > To: ntop [at] listgateway <ntop [at] listgateway>
> > Subject: Re: [Ntop] known double free bug in 4.0.3?
> >
> > On 2011/04/12 08:44, Gary Gatten wrote:
> > > Certainly not an expert, but I see text like "dequeueAddress" and think name/ip lookups. Disable this (-z?) and see how things go. I've had issues with the resolution process that disabling fixed. Maybe some of those same issues aren't fully patched and are impacting you?
> >
> > I would want the TCP session tracking though, this could be useful for
> > accounting and security. How would this affect Netflow accounting?
> >
> > Conor.
> >
> > >
> > > ----- Original Message -----
> > > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > > Sent: Tuesday, April 12, 2011 08:23 AM
> > > To: ntop [at] listgateway <ntop [at] listgateway>
> > > Subject: Re: [Ntop] known double free bug in 4.0.3?
> > >
> > > On 2011/04/12 07:28, Gary Gatten wrote:
> > > > If it dies so often, perhaps try running it with a bunch of stuff disabled; name lookups, decoders, rrd (as much as possible), etc. and see if that makes a difference.
> > >
> > > Gary,
> > >
> > > Error messages below are from Ntop running normally, not in gdb and also
> > > running in gdb + gdb backtrace.
> > >
> > > "While running the main Ntop process in gdb"
> > >
> > > Conor.
> > >
> > > >
> > > > Have you tried running in gdb? I found many of my issues there, except once when it would die every hour running "normal", but running in gdb it would NOT crash! Typical :). The '-k (or -K?)' MAY have had something to do with that.
> > > >
> > > > ----- Original Message -----
> > > > From: Schrodinger [mailto:schrodinger [at] konundrum]
> > > > Sent: Tuesday, April 12, 2011 04:01 AM
> > > > To: ntop [at] listgateway <ntop [at] listgateway>
> > > > Subject: Re: [Ntop] known double free bug in 4.0.3?
> > > >
> > > > Hi,
> > > >
> > > > I have this exact same problem, "double free or corruption". I have
> > > > compiled Ntop with debugging and electric fence and the problem still
> > > > persists. Usually Ntop can't last more than a few hours at a time :( I
> > > > am using Ntop to collect Netflow from different sources via thge native
> > > > Netflow plugin.
> > > >
> > > > I had compiled Ntop with additional debugging to try and locate the
> > > > problem.
> > > >
> > > > [user [at] hos ntop-4.0.3]# export CFLAGS+=-g
> > > > [user [at] hos ntop-4.0.3]# echo $CFLAGS
> > > > -g
> > > > [user [at] hos ntop-4.0.3]# ./autogen.sh --prefix=/usr/local/ntop-4.0.3 --with-memorydebug=2
> > > >
> > > > ===================================================================
> > > >
> > > > This is your ntop 4.0.3 configuration:
> > > >
> > > > Host System Type : x86_64-redhat-linux-gnu
> > > > Preprocessor (cppflags) : -DLINUX -DMEMORY_DEBUG=3 -I/usr/local/include -I/opt/local/include
> > > > Compiler (cflags) : gcc -g -DMEMORY_DEBUG_UNLIMITED -DMUTEX_DEBUG -I/usr/local/include -I/opt/local/include -Wshadow -Wpointer-arith -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fPIC -DPIC
> > > > Defines : -DHAVE_CONFIG_H
> > > > Loader (ldflags) : -L/usr/local/lib -L/opt/local/lib
> > > > Include path :
> > > > System Libs : -lpthread -lcrypt -lc -lssl -lcrypto -lrrd_th -lgdbm -lz -lGeoIP
> > > > Locale : /usr/lib/locale
> > > > Memory Debug : 3
> > > >
> > > > External packages:
> > > >
> > > > LBL pcap .h : standard system headers
> > > > LBL pcap library : standard system libraries
> > > > GNU gdbm .h : standard system headers
> > > > GNU gdbm library : standard system libraries
> > > > zlib .h : standard system headers
> > > > zlib library : standard system libraries
> > > > openSSL .h : standard system headers
> > > > openSSL library : standard system libraries
> > > >
> > > > Install directories:
> > > >
> > > > Default prefix: /usr/local
> > > > Install into: /usr/local/ntop-4.0.3 (default or via --prefix request)
> > > >
> > > > Data files are in /usr/local/ntop-4.0.3/share/ntop
> > > > Config files are in /usr/local/ntop-4.0.3/etc/ntop
> > > > Run directory is /usr/local/ntop-4.0.3/var/ntop
> > > > Plugin files are in /usr/local/ntop-4.0.3/lib/ntop/plugins
> > > > Database files are in /usr/local/ntop-4.0.3/var/ntop
> > > >
> > > > -------------------------------------------------------------------
> > > >
> > > > Here are some logs then that I was able to glean from the system log and/or gdb.
> > > >
> > > > 127.0.0.1 - - [09/Nov/2010:09:11:48 +0000] "GET /MochiKit/Sortable.js HTTP/1.1" 200 83 - - 1
> > > > *** glibc detected *** ./bin/ntop: double free or corruption (fasttop): 0x00002a
> > > > aab41ede00 ***
> > > > ======= Backtrace: =========
> > > > /lib64/libc.so.6[0x2abf39c0430f]
> > > > /lib64/libc.so.6(cfree+0x4b)[0x2abf39c0476b]
> > > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(ntop_safefree+0x16)[0x2abf392de346]
> > > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x3cd)[0x2abf392d36dd]
> > > > /lib64/libpthread.so.0[0x2abf3974473d]
> > > > /lib64/libc.so.6(clone+0x6d)[0x2abf39c65f6d]
> > > >
> > > > While running the main Ntop process in gdb:
> > > >
> > > > Detaching after fork from child process 22963.
> > > > Detaching after fork from child process 22964.
> > > > Detaching after fork from child process 22965.
> > > > Detaching after fork from child process 22966.
> > > > Detaching after fork from child process 22968.
> > > > Detaching after fork from child process 22969.
> > > > Detaching after fork from child process 22970.
> > > > [...]
> > > > Detaching after fork from child process 22971.
> > > > *** glibc detected *** /usr/local/ntop-4.0.3/bin/ntop: double free or corruption (fasttop): 0x00002aaab43181d0 ***
> > > > ======= Backtrace: =========
> > > > /lib64/libc.so.6[0x2aaaab8b930f]
> > > > /lib64/libc.so.6(cfree+0x4b)[0x2aaaab8b976b]
> > > > /usr/local/ntop-4.0.3/lib/libntop-4.0.3.so(dequeueAddress+0x422)[0x2aaaaaf71497]
> > > > /lib64/libpthread.so.0[0x2aaaab3f973d]
> > > > /lib64/libc.so.6(clone+0x6d)[0x2aaaab91af6d]
> > > > ======= Memory map: ========
> > > > 00400000-00410000 r-xp 00000000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > > > 00610000-00611000 rw-p 00010000 68:02 7962633 /usr/local/ntop-4.0.3/bin/ntop
> > > > 00611000-01892000 rw-p 00611000 00:00 0 [heap]
> > > > 40000000-40001000 ---p 40000000 00:00 0
> > > > 40001000-40a01000 rw-p 40001000 00:00 0
> > > > 40a01000-40a02000 ---p 40a01000 00:00 0
> > > > 40a02000-41402000 rw-p 40a02000 00:00 0
> > > > 41402000-41403000 ---p 41402000 00:00 0
> > > > 41403000-41e03000 rw-p 41403000 00:00 0
> > > > 41e03000-41e04000 ---p 41e03000 00:00 0
> > > > 41e04000-42804000 rw-p 41e04000 00:00 0
> > > > 42804000-42805000 ---p 42804000 00:00 0
> > > > 42805000-43205000 rw-p 42805000 00:00 0
> > > > 43205000-43206000 ---p 43205000 00:00 0
> > > > 43206000-43c06000 rw-p 43206000 00:00 0
> > > > 43c06000-43c07000 ---p 43c06000 00:00 0
> > > > [...]
> > > > 2aaaaff4d000-2aaab014c000 ---p 00020000 68:01 163900 /lib64/libexpat.so.0.5.0
> > > > 2aaab014c000-2aaab014f000 rw-p 0001f000 68:01 163900 /lib64/libexpat.so.0.5.0
> > > > 2aaab014f000-2aaab0150000 rw-p 2aaab014f000 00:00 0
> > > > 2aaab0150000-2aaab018b000 r-xp 00000000 68:01 163978 /lib64/libsepol.so.1
> > > > 2aaab018b000-2aaab038b000 ---p 0003b000 68:01 163978 /lib64/libsepol.so.1
> > > > 2aaab038b000-2aaab038c000 rw-p 0003b000 68:01 163978 /lib64/lib
> > > > Program received signal SIGABRT, Aborted.
> > > > [Switching to Thread 0x41e02940 (LWP 22660)]
> > > > 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > > > (gdb)
> > > > (gdb)
> > > >
> > > > (gdb) bt
> > > > #0 0x00002aaaab877265 in raise () from /lib64/libc.so.6
> > > > #1 0x00002aaaab878d10 in abort () from /lib64/libc.so.6
> > > > #2 0x00002aaaab8b184b in __libc_message () from /lib64/libc.so.6
> > > > #3 0x00002aaaab8b930f in _int_free () from /lib64/libc.so.6
> > > > #4 0x00002aaaab8b976b in free () from /lib64/libc.so.6
> > > > #5 0x00002aaaaaf71497 in dequeueAddress (_i=0x0) at address.c:260
> > > > #6 0x00002aaaab3f973d in start_thread () from /lib64/libpthread.so.0
> > > > #7 0x00002aaaab91af6d in clone () from /lib64/libc.so.6
> > > > (gdb)
> > > >
> > > > I am happy to provide any more information you might need or perform any
> > > > additional debugging tasks that would help track this issue down.
> > > >
> > > > Thanks,
> > > > Conor.
> > > > --
> > > > +---------------------------------------------------------------+
> > > > It was a new day yesterday, but it's an old day now.
> > > > MSN: schro5 [at] hotmail
> > > > ICQ: 112562229
> > > > GPG: http://www.konundrum.org/schro.asc
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > <font size="1">
> > > > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > > > </div>
> > > > "This email is intended to be reviewed by only the intended recipient
> > > > and may contain information that is privileged and/or confidential.
> > > > If you are not the intended recipient, you are hereby notified that
> > > > any review, use, dissemination, disclosure or copying of this email
> > > > and its attachments, if any, is strictly prohibited. If you have
> > > > received this email in error, please immediately notify the sender by
> > > > return email and delete this email from your system."
> > > > </font>
> > > >
> > > > _______________________________________________
> > > > Ntop mailing list
> > > > Ntop [at] listgateway
> > > > http://listgateway.unipi.it/mailman/listinfo/ntop
> > >
> > > --
> > > +---------------------------------------------------------------+
> > > It was a new day yesterday, but it's an old day now.
> > > MSN: schro5 [at] hotmail
> > > ICQ: 112562229
> > > GPG: http://www.konundrum.org/schro.asc
> > >
> > >
> > >
> > >
> > >
> > > <font size="1">
> > > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > > </div>
> > > "This email is intended to be reviewed by only the intended recipient
> > > and may contain information that is privileged and/or confidential.
> > > If you are not the intended recipient, you are hereby notified that
> > > any review, use, dissemination, disclosure or copying of this email
> > > and its attachments, if any, is strictly prohibited. If you have
> > > received this email in error, please immediately notify the sender by
> > > return email and delete this email from your system."
> > > </font>
> > >
> > > _______________________________________________
> > > Ntop mailing list
> > > Ntop [at] listgateway
> > > http://listgateway.unipi.it/mailman/listinfo/ntop
> >
> > --
> > +---------------------------------------------------------------+
> > It was a new day yesterday, but it's an old day now.
> > MSN: schro5 [at] hotmail
> > ICQ: 112562229
> > GPG: http://www.konundrum.org/schro.asc
> >
> >
> >
> >
> >
> > <font size="1">
> > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> > </div>
> > "This email is intended to be reviewed by only the intended recipient
> > and may contain information that is privileged and/or confidential.
> > If you are not the intended recipient, you are hereby notified that
> > any review, use, dissemination, disclosure or copying of this email
> > and its attachments, if any, is strictly prohibited. If you have
> > received this email in error, please immediately notify the sender by
> > return email and delete this email from your system."
> > </font>
> >
> > _______________________________________________
> > Ntop mailing list
> > Ntop [at] listgateway
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> --
> +---------------------------------------------------------------+
> It was a new day yesterday, but it's an old day now.
> MSN: schro5 [at] hotmail
> ICQ: 112562229
> GPG: http://www.konundrum.org/schro.asc
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> Ntop [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop

--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>

_______________________________________________
Ntop mailing list
Ntop [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop


schrodinger at konundrum

Apr 15, 2011, 7:32 AM

Post #22 of 22 (2461 views)
Permalink
Re: known double free bug in 4.0.3? [In reply to]

Some additional logs.

Apr 15 14:30:00 host ntop[30329]: LEAK: ERROR: free() - Ptr 0x6fa5710 NOT allocated [@util.c:5893]
Apr 15 14:30:12 host ntop[30329]: LEAK: ERROR: free() - Ptr 0x7236aa0 NOT allocated [@util.c:5893]
Apr 15 14:30:34 host ntop[30329]: LEAK: ERROR: free() - Ptr 0x6f79aa0 NOT allocated [@util.c:5893]
Apr 15 14:30:34 host ntop[30329]: LEAK: ERROR: free() - Ptr 0x6f2a140 NOT allocated [@util.c:5893]
Apr 15 14:30:34 host ntop[30329]: LEAK: ERROR: free() - Ptr 0x75c3cb0 NOT allocated [@util.c:5893]
Apr 15 14:30:34 host ntop[30329]: LEAK: ERROR: free() - Ptr 0x7505290 NOT allocated [@util.c:5893]
[user [at] hos log]# fgrep -v RRD messages| fgrep -c LEAK
46479
[user [at] hos log]#

messages rotated daily.

Conor.
--
+---------------------------------------------------------------+
It was a new day yesterday, but it's an old day now.
MSN: schro5 [at] hotmail
ICQ: 112562229
GPG: http://www.konundrum.org/schro.asc

NTop users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.