Mailing List Archive: DBMail: users

memory leaks in 3.0.2

Index | Next | Previous | View Threaded

pegasus at nerv

Jun 13, 2012, 1:32 AM

Post #1 of 8 (531 views)
Permalink

memory leaks in 3.0.2

Hello,

I'm observing decent memory leaks in all four dbmail daemons. In producton impad grows by about 300MB per day, others less since they're less utilized.

Anyone else observing this?

Is there an easy way to figure out where the memory is leaking without interrupting the service?

--

Jure Pečar
http://jure.pecar.org

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Jun 13, 2012, 1:59 AM

Post #2 of 8 (514 views)
Permalink

Re: memory leaks in 3.0.2 [In reply to]

Am 13.06.2012 10:32, schrieb Jure Pečar:
>
> Hello,
>
> I'm observing decent memory leaks in all four dbmail daemons. In producton impad grows by about 300MB per day, others less since they're less utilized.
>
> Anyone else observing this?
> Is there an easy way to figure out where the memory is leaking without interrupting the service?

this is sadly a know problem easy to reproduce with
5 messages, each a 5 MB jpeg and switch in roundcube
repeatly between this messages in preview

after a short time dbmail-imapd consumes up to 1 GB RAM

i tried to debug this with paul some weeks ago, we found
some memleaks with valgrind, but however the problem still
exists in current HEAD - but be aware using HEAD this time

there is still a bug reintroduced sorting messages in roundcube

"22b964b653ba149b63b9b6787aeac7803c2121e5" is currently
the most bugfree snapshot with working sort and some
improvements compared to 3.0.2 official release

-------- Original-Nachricht --------
Betreff: [Dbmail] sort in roundcubemail broken again
Datum: Mon, 14 May 2012 11:45:42 +0200
Von: Reindl Harald <h.reindl [at] thelounge>
Antwort an: DBMail mailinglist <dbmail [at] dbmail>
Organisation: the lounge interactive design
An: Mailing-List dbmail <dbmail [at] dbmail>

hi

http://git.dbmail.eu/paul/dbmail/commit/?h=dbmail_3_0&id=9e2f272541621d898d9485bfbf43818d7dce708c

in 2da7411789e03a9b51f2e286ea1ea6cc4f3e4303 this seems to be broken again
as long i reset user-preferences and do not touch sort all is fine
but from the moment you are sorting for anything it gets broken

Attachments:

signature.asc (0.26 KB)

harald.leithner at itronic

Jul 13, 2012, 2:28 AM

Post #3 of 8 (451 views)
Permalink

Re: memory leaks in 3.0.2 [In reply to]

Hi Paul,

do we have any updates on this?

Restarted imapd today after 10 Hours with 5GB allocated ram, thats a
problem on a vm with 8GB...

If you need help please tell me how.

thx

Harald

Am 13.06.2012, 10:59 Uhr, schrieb Reindl Harald <h.reindl [at] thelounge>:

>
>
> Am 13.06.2012 10:32, schrieb Jure Pečar:
>>
>> Hello,
>>
>> I'm observing decent memory leaks in all four dbmail daemons. In
>> producton impad grows by about 300MB per day, others less since they're
>> less utilized.
>>
>> Anyone else observing this?
>> Is there an easy way to figure out where the memory is leaking without
>> interrupting the service?
>
> this is sadly a know problem easy to reproduce with
> 5 messages, each a 5 MB jpeg and switch in roundcube
> repeatly between this messages in preview
>
> after a short time dbmail-imapd consumes up to 1 GB RAM
>
> i tried to debug this with paul some weeks ago, we found
> some memleaks with valgrind, but however the problem still
> exists in current HEAD - but be aware using HEAD this time
>
> there is still a bug reintroduced sorting messages in roundcube
>
> "22b964b653ba149b63b9b6787aeac7803c2121e5" is currently
> the most bugfree snapshot with working sort and some
> improvements compared to 3.0.2 official release
>
> -------- Original-Nachricht --------
> Betreff: [Dbmail] sort in roundcubemail broken again
> Datum: Mon, 14 May 2012 11:45:42 +0200
> Von: Reindl Harald <h.reindl [at] thelounge>
> Antwort an: DBMail mailinglist <dbmail [at] dbmail>
> Organisation: the lounge interactive design
> An: Mailing-List dbmail <dbmail [at] dbmail>
>
> hi
>
> http://git.dbmail.eu/paul/dbmail/commit/?h=dbmail_3_0&id=9e2f272541621d898d9485bfbf43818d7dce708c
>
> in 2da7411789e03a9b51f2e286ea1ea6cc4f3e4303 this seems to be broken again
> as long i reset user-preferences and do not touch sort all is fine
> but from the moment you are sorting for anything it gets broken
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

paul at nfg

Aug 1, 2012, 3:52 AM

Post #4 of 8 (428 views)
Permalink

Re: memory leaks in 3.0.2 [In reply to]

On 07/13/2012 11:28 AM, Harald Leithner wrote:
> Hi Paul,
>
> do we have any updates on this?

Not much. The problem is very difficult to reproduce reliably. Valgrind
traces generated using the steps outlined by Harald Reindl below seem to
point in the direction of either glib's slice-allocator, or thread-races
in dbmail that I'm unable to explain.

>>> I'm observing decent memory leaks in all four dbmail daemons. In
>>> producton impad grows by about 300MB per day, others less since
>>> they're less utilized.

Any leakage in lmtp/pop3/timsieve should be *much* easier to debug and
fix since those are *not* threaded. If anyone is up to present me with
command sequences that lead to non-trivial leakage I'd be happy to look
into it.

>>>
>>> Anyone else observing this?
>>> Is there an easy way to figure out where the memory is leaking
>>> without interrupting the service?

No. You need to:

compile with debug-flags (-j) and without stripping the binaries
run the deamons using valgrind, i.e.

G_SLICE=always-malloc valgrind --leak-check=full
--log-file=/tmp/vg-lmtpd.log .../dbmail-lmtpd -D

keep it running for a while, stop the service (ctrl-c), study the logs.

Any indication in the logs of definite leakage (ignore possible or
indirect leakage) definitely needs to be fixed if it originates in the
dbmail code.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

paul at nfg

Aug 6, 2012, 6:11 AM

Post #5 of 8 (417 views)
Permalink

Re: memory leaks in 3.0.2 [In reply to]

On 06/13/2012 10:59 AM, Reindl Harald wrote:

>> Anyone else observing this?
>> Is there an easy way to figure out where the memory is leaking without interrupting the service?

As for roundcube this appears to be alleviated by using

$rcmail_config['enable_caching'] = TRUE;

> there is still a bug reintroduced sorting messages in roundcube

I'm still unable to reproduce this. However, while testing for this one,
I noticed that using:

$rcmail_config['index_sort'] = FALSE;

will trigger a server-side SORT which is quite slow. I suggest using

$rcmail_config['index_sort'] = TRUE;

which will cause much faster client-side sorting

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

harald.leithner at itronic

Aug 6, 2012, 6:34 AM

Post #6 of 8 (422 views)
Permalink

Re: memory leaks in 3.0.2 [In reply to]

Paul,

which version of roundcube do you use? the option:
$rcmail_config['index_sort']
has been removed in version 0.4-beta.
from the changelog:
- Removed 'index_sort' option, now we're using empty 'message_sort_col'
for this

also enable_caching has changed in Version 0.6:
- Option enable_caching replaced by imap_cache and messages_cache options

thx for trying to fix this issue.

Harald Leithner

Am 06.08.2012, 15:11 Uhr, schrieb Paul J Stevens <paul [at] nfg>:

> On 06/13/2012 10:59 AM, Reindl Harald wrote:
>
>>> Anyone else observing this?
>>> Is there an easy way to figure out where the memory is leaking without
>>> interrupting the service?
>
> As for roundcube this appears to be alleviated by using
>
> $rcmail_config['enable_caching'] = TRUE;
>
>> there is still a bug reintroduced sorting messages in roundcube
>
> I'm still unable to reproduce this. However, while testing for this one,
> I noticed that using:
>
> $rcmail_config['index_sort'] = FALSE;
>
> will trigger a server-side SORT which is quite slow. I suggest using
>
> $rcmail_config['index_sort'] = TRUE;
>
> which will cause much faster client-side sorting
>
>
>

--
Harald Leithner

ITronic
Vogelweidplatz 12, 1150 Wien, Austria
Tel: +43-1-786 23 88
Fax: +43-1-98 52 077
Mobil: +43-669-123 78 4 78
Mail: leithner [at] itronic | itronic.at
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Aug 6, 2012, 6:44 AM

Post #7 of 8 (418 views)
Permalink

Re: memory leaks in 3.0.2 [In reply to]

Hi Paul

Am 06.08.2012 15:11, schrieb Paul J Stevens:
> On 06/13/2012 10:59 AM, Reindl Harald wrote:
>
>>> Anyone else observing this?
>>> Is there an easy way to figure out where the memory is leaking without interrupting the service?
>
> As for roundcube this appears to be alleviated by using
>
> $rcmail_config['enable_caching'] = TRUE;

only a workaround which leads to have a large amount
of data which origins in a database in another one

i doubt roundcube is only one way to trigger this

>> there is still a bug reintroduced sorting messages in roundcube
>
> I'm still unable to reproduce this. However, while testing for this one

there was a reply from you some weeks ago that you reverted the fix
because it broke postgres and there is another one which needs a scheme
change - currently i use 22b964b653ba149b63b9b6787aeac7803c2121e5 which
has the old one and the bulk/list-header changes for autoreplies

see Fri, 06 Apr 2012 22:27:50 +0200 [Dbmail] fixed: sort in roundcube (and maybe other clients)

because the current HEAD has a lot of changes i hestitate it in production
3.0.3 with stable changes and clear documentation about the scheme-change
for correct sort-order would be really fine sooner or later

> I noticed that using:
> $rcmail_config['index_sort'] = FALSE;
> will trigger a server-side SORT which is quite slow. I suggest using
> $rcmail_config['index_sort'] = TRUE;
> which will cause much faster client-side sorting

for which roundcube version is this?
i do not find this value in my config

however, the problem did also exist on the iPhone of the
same user after change sorting to a different column
_______________________

BTW:

http://git.dbmail.eu/paul/dbmail/commit/?id=bd7e9e1687a801d9be93dc71415671918e5f97f1
IMAP: fix broken structure for Apple multipart disposition

seems to work with the test-message with the missing attachment
hopefully this is not too specific.....

Regards
Harry

Attachments:

signature.asc (0.26 KB)

paul at nfg

Aug 6, 2012, 8:15 AM

Post #8 of 8 (418 views)
Permalink

Re: memory leaks in 3.0.2 [In reply to]

On 08/06/2012 03:44 PM, Reindl Harald wrote:

>> $rcmail_config['enable_caching'] = TRUE;
>
> only a workaround which leads to have a large amount
> of data which origins in a database in another one
>
> i doubt roundcube is only one way to trigger this

If you find one, please let me know. The leakage seems to occur if and
only when you retrieve the same message simultaneously in concurrently
connected clients. So yes: it will happen, but that also makes it *very*
difficult to reproduce - other than with roundcube. I've tried with
imaptest which allows you to setup multiple client session. But I
couldn't trigger the problem: zero-leakage.

>> I noticed that using:
>> $rcmail_config['index_sort'] = FALSE;
>> will trigger a server-side SORT which is quite slow. I suggest using
>> $rcmail_config['index_sort'] = TRUE;
>> which will cause much faster client-side sorting
>
> for which roundcube version is this?
> i do not find this value in my config

an older one (0.3.1) from ubuntu-lucid

> however, the problem did also exist on the iPhone of the
> same user after change sorting to a different column

Mmm, my wifi network is flaky, so I can't test with my iPad.

d9be93dc71415671918e5f97f1
> IMAP: fix broken structure for Apple multipart disposition
>
> seems to work with the test-message with the missing attachment
> hopefully this is not too specific.....

No, it wasn't.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads