Mailing List Archive: DBMail: users

imapd and overquota

Index | Next | Previous | View Threaded

pegasus at nerv

Apr 10, 2012, 8:37 AM

Post #1 of 12 (540 views)
Permalink

imapd and overquota

Hello all,

I see there are some dbmail and roundcube users on this list.

I'm deploying dbmail+roundcube+dbmailadmin solution and am in the final stages of testing. There are still some interesting bugs to sort out.

For example, if I create a mailbox with artificially small quota (say, 1MB) to easily test near and over quota scenarios, this is what happen: when appending mail to a 99% full mailbox that would put it over quota, roundcube freezes.

Looking into what's going on I discovered this in imap protocol dialog:

C: A0003 LSUB "" Sent
S: * LSUB (\hasnochildren) "/" "Sent"
S: A0003 OK LSUB completed
C: A0004 LIST "" Sent
S: * LIST (\hasnochildren) "/" "Sent"
S: A0004 OK LIST completed
C: A0005 APPEND Sent (\Seen) {12824}
S: + OK
C: MIME-Version: 1.0
<content of mail>
<no reply from server>

So roundcube is right in freezing, since it waits for a reply from imap server that never comes. Looking at this protocol dialog, server is misleading the client when it replies OK to an APPEND command, where it should actually reject it (it already has the size info and can calculate it would cause overquota condition).

This is on fresh dbmail 3.0.2 with just a few test users and roundcube 0.8-beta.

Any hints before I open a bug report for this?

--

Jure Pečar
http://jure.pecar.org
http://f5j.eu
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

paul at nfg

Apr 11, 2012, 1:42 AM

Post #2 of 12 (518 views)
Permalink

Re: imapd and overquota [In reply to]

On 04/10/2012 05:37 PM, Jure Pečar wrote:

> C: A0005 APPEND Sent (\Seen) {12824}
> S: + OK
> C: MIME-Version: 1.0
> <content of mail>
> <no reply from server>

> So roundcube is right in freezing, since it waits for a reply from
> imap server that never comes. Looking at this protocol dialog, server
> is misleading the client when it replies OK to an APPEND command,
> where it should actually reject it (it already has the size info and
> can calculate it would cause overquota condition).

The server is not misleading with the + OK. That just means 'ok, ready
to receive data', not 'ok, ready to receive data and ready to store
message'.

However, in this case the server should have sent a

S: A0005 BAD quota violation

response.

Please file a bug, if you will.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

paul at nfg

Apr 11, 2012, 1:51 AM

Post #3 of 12 (518 views)
Permalink

Re: imapd and overquota [In reply to]

On 04/11/2012 10:42 AM, Paul J Stevens wrote:

> Please file a bug, if you will.

No need. Fixed already.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Apr 11, 2012, 6:00 AM

Post #4 of 12 (521 views)
Permalink

Re: imapd and overquota [In reply to]

Am 11.04.2012 10:51, schrieb Paul J Stevens:
> On 04/11/2012 10:42 AM, Paul J Stevens wrote:
>
>> Please file a bug, if you will.
>
> No need. Fixed already.

does this also fix the (reintroduced) problem to stop
if a client tries to store a > 500 MB message in the
dafts folder?

i tried this yet and dbmail-imapd starts to consume hughe
memory and CPU and is no longer responsible for any
client

we noticed this the first time last year as our frontoffice
took the wrong attachment (2 GB) and AppleMail started
to store it in "drafts" while dbmail crashed permanently
and everytime it came back AppleMail treid again

the hughe memory usage is here the biggest problem because
this leads sooner or later in OOM-killer killing mysqld
the hard way :-(

Attachments:

signature.asc (0.26 KB)

paul at nfg

Apr 11, 2012, 6:13 AM

Post #5 of 12 (517 views)
Permalink

Re: imapd and overquota [In reply to]

paul at nfg

Apr 11, 2012, 8:09 AM

Post #6 of 12 (516 views)
Permalink

Re: imapd and overquota [In reply to]

On 04/11/2012 03:00 PM, Reindl Harald wrote:
>
>
> Am 11.04.2012 10:51, schrieb Paul J Stevens:
>> On 04/11/2012 10:42 AM, Paul J Stevens wrote:
>>
>>> Please file a bug, if you will.
>>
>> No need. Fixed already.
>
> does this also fix the (reintroduced) problem to stop
> if a client tries to store a > 500 MB message in the
> dafts folder?
>
> i tried this yet and dbmail-imapd starts to consume hughe
> memory and CPU and is no longer responsible for any
> client

After testing with 100MB size message appends I came up with:

http://git.dbmail.eu/paul/dbmail/commit/?id=4b095dbcb956aa3538b21912ba34b0db088e4967

which will improve the situation, but *not* fix it completely.

The OOM situation is caused by excessive copying:

- from the network to a receive buffer
- from the receive buffer to the token buffer
- from the token buffer to the append command's argument list

and that's just from the imap network handler. After this the string in
the argument list is converted into a DbmailMessage object, which leads
to even more copying.

Above patch will improve the CPU load significantly, but not the copying...

Maybe we should set a configurable limit on APPEND message sizes. No use
allowing 1GB messages on APPEND if your MTA doesn't allow messages
larger than 100MB for example.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Apr 11, 2012, 8:19 AM

Post #7 of 12 (513 views)
Permalink

Re: imapd and overquota [In reply to]

Am 11.04.2012 17:09, schrieb Paul J Stevens:

> After testing with 100MB size message appends I came up with:
> http://git.dbmail.eu/paul/dbmail/commit/?id=4b095dbcb956aa3538b21912ba34b0db088e4967
> which will improve the situation, but *not* fix it completely.
>
> The OOM situation is caused by excessive copying:
>
> - from the network to a receive buffer
> - from the receive buffer to the token buffer
> - from the token buffer to the append command's argument list
>
> and that's just from the imap network handler. After this the string in
> the argument list is converted into a DbmailMessage object, which leads
> to even more copying.
>
> Above patch will improve the CPU load significantly, but not the copying...
>
> Maybe we should set a configurable limit on APPEND message sizes. No use
> allowing 1GB messages on APPEND if your MTA doesn't allow messages
> larger than 100MB for example

that would be perfect and trigger the error before the
composer has finsihed and press "send"

over WAN this is not so problematic (but well, i have seen
people upload 300 MB files over 1 Mbit), it is growing
by more and more users with faster internet (in my case at
home 100/10 Mbit)

over Gigabit LAN it kills the server within seconds

Attachments:

signature.asc (0.26 KB)

h.reindl at thelounge

Apr 11, 2012, 8:39 AM

Post #8 of 12 (519 views)
Permalink

Re: imapd and overquota [In reply to]

Am 11.04.2012 17:19, schrieb Reindl Harald:
>
>
> Am 11.04.2012 17:09, schrieb Paul J Stevens:
>
>> After testing with 100MB size message appends I came up with:
>> http://git.dbmail.eu/paul/dbmail/commit/?id=4b095dbcb956aa3538b21912ba34b0db088e4967
>> which will improve the situation, but *not* fix it completely.
>>
>> The OOM situation is caused by excessive copying:
>>
>> - from the network to a receive buffer
>> - from the receive buffer to the token buffer
>> - from the token buffer to the append command's argument list
>>
>> and that's just from the imap network handler. After this the string in
>> the argument list is converted into a DbmailMessage object, which leads
>> to even more copying.
>>
>> Above patch will improve the CPU load significantly, but not the copying...
>>
>> Maybe we should set a configurable limit on APPEND message sizes. No use
>> allowing 1GB messages on APPEND if your MTA doesn't allow messages
>> larger than 100MB for example
>
> that would be perfect and trigger the error before the
> composer has finsihed and press "send"
>
> over WAN this is not so problematic (but well, i have seen
> people upload 300 MB files over 1 Mbit), it is growing
> by more and more users with faster internet (in my case at
> home 100/10 Mbit)
>
> over Gigabit LAN it kills the server within seconds

Out of memory: Kill process 23371 (dbmail-imapd) score 882 or sacrifice child
Killed process 23371 (dbmail-imapd) total-vm:55292878000kB, anon-rss:14583608kB, file-rss:0kB

see htop-screenshot while 400 MB attachment is saved in drafts
dbmail-3.0.2-1334157721.fc16.20120411.rh.4b095dbcb956aa3538b21912ba34b0db088e4967.x86_64

the machine is a Quad Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz with 16 GB RAM
after some time it started heavily to swap and get compleltly unrepsposible
before OOM-killer shot down dbmail-imapd, this is heavy :-)

Attachments:

screen.png (250 KB)

signature.asc (0.26 KB)

pegasus at nerv

Apr 11, 2012, 9:13 AM

Post #9 of 12 (514 views)
Permalink

Re: imapd and overquota [In reply to]

On Wed, 11 Apr 2012 10:51:58 +0200
Paul J Stevens <paul [at] nfg> wrote:

> On 04/11/2012 10:42 AM, Paul J Stevens wrote:
>
> > Please file a bug, if you will.
>
> No need. Fixed already.

Confirming that the fix works.

--

Jure Pečar
http://jure.pecar.org
http://f5j.eu
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

paul at nfg

Apr 12, 2012, 1:07 AM

Post #10 of 12 (512 views)
Permalink

Re: imapd and overquota [In reply to]

On 04/11/2012 05:19 PM, Reindl Harald wrote:

> that would be perfect and trigger the error before the
> composer has finsihed and press "send"

Added in patch:

http://git.dbmail.eu/paul/dbmail/commit/?id=e8edcb868150da613fb2fd89fece0956fc7c93f0

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Apr 12, 2012, 1:39 AM

Post #11 of 12 (513 views)
Permalink

Re: imapd and overquota [In reply to]

Am 12.04.2012 10:07, schrieb Paul J Stevens:
> On 04/11/2012 05:19 PM, Reindl Harald wrote:
>
>> that would be perfect and trigger the error before the
>> composer has finsihed and press "send"
>
>
> Added in patch:
>
> http://git.dbmail.eu/paul/dbmail/commit/?id=e8edcb868150da613fb2fd89fece0956fc7c93f0

thank you

testserver started build, considering deploy after some tests today
is the value in bytes like postfix's "message_size_limit = 36700160"?

Attachments:

signature.asc (0.26 KB)

paul at nfg

Apr 12, 2012, 2:04 AM

Post #12 of 12 (519 views)
Permalink

Re: imapd and overquota [In reply to]

On 04/12/2012 10:39 AM, Reindl Harald wrote:

> testserver started build, considering deploy after some tests today
> is the value in bytes like postfix's "message_size_limit = 36700160"?

Yes, so *no* 'magic' interpretation of M, G, or other postfixes.

You *can* use: base-10, hexadecimal, and octal numbers though, since I'm
using strtoul(2) to convert the string.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads