Mailing List Archive: DBMail: users

Outlook 2013 Imap implementation is broken

1 2

View All

Index | Next | Previous | View Threaded

harald.leithner at itronic

Jul 18, 2013, 2:27 AM

Post #1 of 28 (256 views)
Permalink

Outlook 2013 Imap implementation is broken

Hi,

if you have the chance to suggest a Mail product prevent him to buy
outlook 2013 because the IMAP Support is really broken.

I have the problem on multiple Outlook 2013 users getting there mails
delayed, this could be up to 8 hours.

On Webmail the mail is there in Outlook 2013 doesn't appear, I didn't
traced the problem atm, but there are many reports about sync problems in
the net.

Maybe anyone knows a workaround or a fix?

--
Harald Leithner

ITronic
Vogelweidplatz 12, 1150 Wien, Austria
Tel: +43-1-786 23 88
Fax: +43-1-98 52 077
Mobil: +43-699-123 78 4 78
Mail: leithner [at] itronic | itronic.at
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

jake at vapourforge

Jul 18, 2013, 2:33 AM

Post #2 of 28 (254 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

On 18/07/13 19:27, Harald Leithner wrote:
> Hi,
>
> if you have the chance to suggest a Mail product prevent him to buy
> outlook 2013 because the IMAP Support is really broken.
>
> I have the problem on multiple Outlook 2013 users getting there mails
> delayed, this could be up to 8 hours.
>
> On Webmail the mail is there in Outlook 2013 doesn't appear, I didn't
> traced the problem atm, but there are many reports about sync problems
> in the net.
>
> Maybe anyone knows a workaround or a fix?
>
+1 to this
If you use roaming profiles its *totally* broken, as soon as your user
roams with IMAP their email will stop working totally, sending fails,
receiving goes stupid. They have known about this for years, their
solution is "use exchange"
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Jul 18, 2013, 2:33 AM

Post #3 of 28 (250 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 18.07.2013 11:27, schrieb Harald Leithner:
> if you have the chance to suggest a Mail product prevent him to buy outlook 2013
> because the IMAP Support is really broken

sadly i am not in the position to suggest what a new customer has rolled out in the past :-)

however, the current HEAD of dbmail has a lot of more troubles with
Outlook 2013 as 22b964b653ba149b63b9b6787aeac7803c2121e5 which i am
still running because i am not sure which later version i can expect
to not introduce new troubles or bring some old from the debug
sessions with Paul in 2012 back :-(

> I have the problem on multiple Outlook 2013 users getting there mails delayed, this could be up to 8 hours.

my problem with the current HEAD is that there isd no chance to see
new mails at all and if you move them with Outlook 2013 itself in a
different folder you will never see them again :-)

> On Webmail the mail is there in Outlook 2013 doesn't appear, I didn't traced
> the problem atm, but there are many reports about sync problems in the net.
>
> Maybe anyone knows a workaround or a fix?

however, it must happen on dbmail side, sad but true
even if there would be a magical Outlook setting it would be
a support nightmare und you never know if a customer reinstalled
his machine or created a new user or whatever wothout some
non default setting

Attachments:

signature.asc (0.26 KB)

harald.leithner at itronic

Jul 18, 2013, 2:40 AM

Post #4 of 28 (250 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 18.07.2013, 11:33 Uhr, schrieb Reindl Harald <h.reindl [at] thelounge>:

>
>
> Am 18.07.2013 11:27, schrieb Harald Leithner:
>> if you have the chance to suggest a Mail product prevent him to buy
>> outlook 2013
>> because the IMAP Support is really broken
>
> sadly i am not in the position to suggest what a new customer has rolled
> out in the past :-)

;-)

>
> however, the current HEAD of dbmail has a lot of more troubles with
> Outlook 2013 as 22b964b653ba149b63b9b6787aeac7803c2121e5 which i am
> still running because i am not sure which later version i can expect
> to not introduce new troubles or bring some old from the debug
> sessions with Paul in 2012 back :-(

No problem so far except Outlook 2013 and double mails and dying pop3d and
memory leak and hmm I think thats all atm.

>
>> I have the problem on multiple Outlook 2013 users getting there mails
>> delayed, this could be up to 8 hours.
>
> my problem with the current HEAD is that there isd no chance to see
> new mails at all and if you move them with Outlook 2013 itself in a
> different folder you will never see them again :-)

This seam to work for the customer, he has many folders ;-)

>
>> On Webmail the mail is there in Outlook 2013 doesn't appear, I didn't
>> traced
>> the problem atm, but there are many reports about sync problems in the
>> net.
>>
>> Maybe anyone knows a workaround or a fix?
>
> however, it must happen on dbmail side, sad but true
> even if there would be a magical Outlook setting it would be
> a support nightmare und you never know if a customer reinstalled
> his machine or created a new user or whatever wothout some
> non default setting
>

there are also problems with gmail and godaddy and 1und1 so the problem is
in outlook and hopefully not in a setting ;-)

--
Harald Leithner

ITronic
Vogelweidplatz 12, 1150 Wien, Austria
Tel: +43-1-786 23 88
Fax: +43-1-98 52 077
Mobil: +43-699-123 78 4 78
Mail: leithner [at] itronic | itronic.at
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

paul at nfg

Jul 18, 2013, 3:22 AM

Post #5 of 28 (249 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

On 07/18/2013 11:40 AM, Harald Leithner wrote:

> No problem so far except Outlook 2013 and double mails and dying pop3d
> and memory leak and hmm I think thats all atm.

What memory leaks! Those have been fixed. A growing memory foot-print is
not a direct indication of a memory leak.

Same for the pop3 crashes: fixed as far as I know. I've pushed that one
last Tuesday. I can't create crashes anymore, and I've kept 20
concurrent clients trying to crash things for over an hour, doing
thousands of commands.

The duplicate emails is rather new so I don't know.

As far as Outlook 2013 is concerned, I don't have it, so I can't test
it. A wireshark session dump would be helpful, but don't spam me with
megabytes of useless dumps. Maybe Outlook is issuing commands dbmail
doesn't (yet) support.

> there are also problems with gmail and godaddy and 1und1 so the problem
> is in outlook and hopefully not in a setting ;-)

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

harald.leithner at itronic

Jul 18, 2013, 3:30 AM

Post #6 of 28 (250 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 18.07.2013, 12:22 Uhr, schrieb Paul J Stevens <paul [at] nfg>:

> On 07/18/2013 11:40 AM, Harald Leithner wrote:
>
>> No problem so far except Outlook 2013 and double mails and dying pop3d
>> and memory leak and hmm I think thats all atm.

That wasn't a attack on you sorry if you feel like this.

>
> What memory leaks! Those have been fixed. A growing memory foot-print is
> not a direct indication of a memory leak.

My Imapd got killed yesterday at 1GB Memory after running 2 days I think,
last week it was killed at 1.8 GB
If this usage is normal then I will change my monitoring settings.

>
> Same for the pop3 crashes: fixed as far as I know. I've pushed that one
> last Tuesday. I can't create crashes anymore, and I've kept 20
> concurrent clients trying to crash things for over an hour, doing
> thousands of commands.

I'm running 6e8d737299998f6df55757e75cd3afe0bd7c440f I think it crashed
yeasterday but I'm not sure.

>
> The duplicate emails is rather new so I don't know.
>
No I will try to debug this.

> As far as Outlook 2013 is concerned, I don't have it, so I can't test
> it. A wireshark session dump would be helpful, but don't spam me with
> megabytes of useless dumps. Maybe Outlook is issuing commands dbmail
> doesn't (yet) support.

If I have time I will try to debug this too, but I hope MS fixes this
before I start debugging.

>
>> there are also problems with gmail and godaddy and 1und1 so the problem
>> is in outlook and hopefully not in a setting ;-)
>

and thanks again you doing a great work!

--
Harald Leithner

ITronic
Vogelweidplatz 12, 1150 Wien, Austria
Tel: +43-1-786 23 88
Fax: +43-1-98 52 077
Mobil: +43-699-123 78 4 78
Mail: leithner [at] itronic | itronic.at
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Jul 18, 2013, 3:42 AM

Post #7 of 28 (255 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 18.07.2013 12:22, schrieb Paul J Stevens:
> As far as Outlook 2013 is concerned, I don't have it, so I can't test
> it. A wireshark session dump would be helpful, but don't spam me with
> megabytes of useless dumps. Maybe Outlook is issuing commands dbmail
> doesn't (yet) support

unlikely

snapshot from last year 22b964b653ba149b63b9b6787aeac7803c2121e5 does
not have the problem - i made a bugreport with a debug-log as well
sent it to the list!

Attachments:

signature.asc (0.26 KB)

paul at nfg

Jul 18, 2013, 4:37 AM

Post #8 of 28 (243 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

On 07/18/2013 12:30 PM, Harald Leithner wrote:
>> What memory leaks! Those have been fixed. A growing memory foot-print is
>> not a direct indication of a memory leak.
>
> My Imapd got killed yesterday at 1GB Memory after running 2 days I
> think, last week it was killed at 1.8 GB
> If this usage is normal then I will change my monitoring settings.

Growing memory usage is a problem. Long running imap sessions may do
that. I think I've done just about as much as I can about that, given
the current single process design.

Things to try:

- Run with and without G_SLICE=all which disables the Glib slice
allocator in favor of the system libc's malloc. Myself, I run without
G_SLICE=all, so *with* the slice allocator.

- Try jemalloc by installing libjemalloc-dev and running
./configure --with-jemalloc

- I've also implemented segmented memory arenas, much like apache's
request-local memory pools. I used it to create imap-session specific
memory pools. But since sessions can quite commonly be long-running,
this just made things worse. Still, the memory pool infrastructure is
there, and might help if I use pools that last just the duration of a
imap command - which are mostly short running.

That said, 1.8GB quite a lot! But is that RSS or VSS?

I use a simple RDD-tool grapher to keep an eye on this. I typically see
a VSS around around 1GB - dropped to 500MB since around two weeks.

RSS is always much lower: around 200MB

Both should be pretty much flat.

>> Same for the pop3 crashes: fixed as far as I know. I've pushed that one
>> last Tuesday. I can't create crashes anymore, and I've kept 20
>> concurrent clients trying to crash things for over an hour, doing
>> thousands of commands.
>
> I'm running 6e8d737299998f6df55757e75cd3afe0bd7c440f I think it crashed
> yeasterday but I'm not sure.

Correct. That one didn't quite nail it. The next one did, afaik. Better
yet b2958774333d81a441061b2a36fd1f4ae8a45562 was the last revision from
that sprint.

> If I have time I will try to debug this too, but I hope MS fixes this
> before I start debugging.

I'm not even sure it's a MS bug, or just a very particular way of
reading the RFC's. From what I read, they are very much after full
compliance.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

harald.leithner at itronic

Jul 18, 2013, 4:54 AM

Post #9 of 28 (242 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 18.07.2013, 13:37 Uhr, schrieb Paul J Stevens <paul [at] nfg>:

> On 07/18/2013 12:30 PM, Harald Leithner wrote:
>>> What memory leaks! Those have been fixed. A growing memory foot-print
>>> is
>>> not a direct indication of a memory leak.
>>
>> My Imapd got killed yesterday at 1GB Memory after running 2 days I
>> think, last week it was killed at 1.8 GB
>> If this usage is normal then I will change my monitoring settings.
>
> Growing memory usage is a problem. Long running imap sessions may do
> that. I think I've done just about as much as I can about that, given
> the current single process design.
>
> Things to try:
>
> - Run with and without G_SLICE=all which disables the Glib slice
> allocator in favor of the system libc's malloc. Myself, I run without
> G_SLICE=all, so *with* the slice allocator.
>
> - Try jemalloc by installing libjemalloc-dev and running
> ./configure --with-jemalloc
>
> - I've also implemented segmented memory arenas, much like apache's
> request-local memory pools. I used it to create imap-session specific
> memory pools. But since sessions can quite commonly be long-running,
> this just made things worse. Still, the memory pool infrastructure is
> there, and might help if I use pools that last just the duration of a
> imap command - which are mostly short running.
>
> That said, 1.8GB quite a lot! But is that RSS or VSS?
>
> I use a simple RDD-tool grapher to keep an eye on this. I typically see
> a VSS around around 1GB - dropped to 500MB since around two weeks.
>
> RSS is always much lower: around 200MB
>
> Both should be pretty much flat.

Atm running since 2 Hourse:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24551 dbmail 20 0 1407m 763m 3944 S 0.7 2.4 5:07.73 dbmail-imapd

>
>>> Same for the pop3 crashes: fixed as far as I know. I've pushed that one
>>> last Tuesday. I can't create crashes anymore, and I've kept 20
>>> concurrent clients trying to crash things for over an hour, doing
>>> thousands of commands.
>>
>> I'm running 6e8d737299998f6df55757e75cd3afe0bd7c440f I think it crashed
>> yeasterday but I'm not sure.
>
> Correct. That one didn't quite nail it. The next one did, afaik. Better
> yet b2958774333d81a441061b2a36fd1f4ae8a45562 was the last revision from
> that sprint.

compiled HEAD with --with-jemalloc now

>
>> If I have time I will try to debug this too, but I hope MS fixes this
>> before I start debugging.
>
> I'm not even sure it's a MS bug, or just a very particular way of
> reading the RFC's. From what I read, they are very much after full
> compliance.
>
>

hmm... good or bad?

--
Harald Leithner

ITronic
Vogelweidplatz 12, 1150 Wien, Austria
Tel: +43-1-786 23 88
Fax: +43-1-98 52 077
Mobil: +43-699-123 78 4 78
Mail: leithner [at] itronic | itronic.at
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

paul at nfg

Jul 19, 2013, 12:42 AM

Post #10 of 28 (238 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

>> I use a simple RDD-tool grapher to keep an eye on this. I typically see
>> a VSS around around 1GB - dropped to 500MB since around two weeks.

I was thinking back at what might have caused the drop in virtual memory
usage. Nothing in the dbmail codebase changed that would explain this,
but I did go from linux 3.5.1 to 3.8.0

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

paul at nfg

Jul 22, 2013, 7:15 AM

Post #11 of 28 (219 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

On 18-07-13 12:42, Reindl Harald wrote:
>
>
> Am 18.07.2013 12:22, schrieb Paul J Stevens:
>> As far as Outlook 2013 is concerned, I don't have it, so I can't
>> test it. A wireshark session dump would be helpful, but don't
>> spam me with megabytes of useless dumps. Maybe Outlook is issuing
>> commands dbmail doesn't (yet) support
>
> unlikely
>
> snapshot from last year 22b964b653ba149b63b9b6787aeac7803c2121e5
> does not have the problem - i made a bugreport with a debug-log as
> well sent it to the list!

Someone provided me with a Outlook 2013 VPS yesterday, so I was able
to test and fix that version this morning, as far as I can tell.

Perhaps someone has a Outlook 2007 (vps?) available somewhere I can
connect to?

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Jul 22, 2013, 8:33 AM

Post #12 of 28 (219 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 22.07.2013 16:15, schrieb Paul J Stevens:
> On 18-07-13 12:42, Reindl Harald wrote:
>>
>>
>> Am 18.07.2013 12:22, schrieb Paul J Stevens:
>>> As far as Outlook 2013 is concerned, I don't have it, so I can't
>>> test it. A wireshark session dump would be helpful, but don't
>>> spam me with megabytes of useless dumps. Maybe Outlook is issuing
>>> commands dbmail doesn't (yet) support
>>
>> unlikely
>>
>> snapshot from last year 22b964b653ba149b63b9b6787aeac7803c2121e5
>> does not have the problem - i made a bugreport with a debug-log as
>> well sent it to the list!
>
> Someone provided me with a Outlook 2013 VPS yesterday, so I was able
> to test and fix that version this morning, as far as I can tell.

thank you - sadly i have no 2013 version but good to know it's fine

> Perhaps someone has a Outlook 2007 (vps?) available somewhere I can
> connect to?

http://git.dbmail.eu/paul/dbmail/commit/?id=1364b122de3b7e624656b3d25098abfae6fd6a30
i tested HEAD with Outlook 2007 and still the same problem :-(
2003/2010 are fine

i have no way to offer the test VM with Outlook 2007
it's a regsitered company license :-(

Attachments:

signature.asc (0.26 KB)

paul at nfg

Jul 22, 2013, 11:47 AM

Post #13 of 28 (219 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

On 07/22/2013 05:33 PM, Reindl Harald wrote:

> http://git.dbmail.eu/paul/dbmail/commit/?id=1364b122de3b7e624656b3d25098abfae6fd6a30
> i tested HEAD with Outlook 2007 and still the same problem :-(
> 2003/2010 are fine

I just tested with Outlook 2007 (Thanks Jorge!) and copy/paste or
dragging messages to another mailbox works just fine. Weird thing is I
can't reproduce your problem, not on today's snapshot, not on last month's.

Can you remove and re-add the account? Seems to be standard Microsoft
practice if the mailboxes are out of sync somehow.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Jul 23, 2013, 3:55 AM

Post #14 of 28 (214 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 22.07.2013 20:47, schrieb Paul J Stevens:
> On 07/22/2013 05:33 PM, Reindl Harald wrote:
>> http://git.dbmail.eu/paul/dbmail/commit/?id=1364b122de3b7e624656b3d25098abfae6fd6a30
>> i tested HEAD with Outlook 2007 and still the same problem :-(
>> 2003/2010 are fine
>
> I just tested with Outlook 2007 (Thanks Jorge!) and copy/paste or
> dragging messages to another mailbox works just fine. Weird thing is I
> can't reproduce your problem, not on today's snapshot, not on last month's.
>
> Can you remove and re-add the account? Seems to be standard Microsoft
> practice if the mailboxes are out of sync somehow

did i say that i hate Microsoft?
indeed - after delete the account it works
seems that one of the in the meantime fixed bugs confused it enduring
______________________________

a small problem seems to exist in the 3.1.0 build affecting
Thunderbird as well as all Outllok versions after running a longer
time and move messages out of a folder, randomly switch to a empty
folder in Outlook results in "IMAP command failed" and thunderbird
says "server answered invalid sequence"

i try to get one of the clients triggering this again and get
a wireshark dump what happens in the communication, this may
get annoying for users

the "invalid sequence" is mostly triggerable by "empty trash"
with https://addons.mozilla.org/en-us/thunderbird/addon/xpunge/
if you have it configured to expunge any folders and empty all
trashes
______________________________

what scares me currently more is this below

if users get contents of other users this is major critical
until now i never seens this, but i hestitate currently
to consider a upgrade on the production machines

-------- Original-Nachricht --------
Betreff: Re: [Dbmail] Wrong E-Mail Content with imap
Datum: Mon, 22 Jul 2013 14:11:25 +0200
Von: Harald Leithner <harald.leithner [at] itronic>
Antwort an: DBMail mailinglist <dbmail [at] dbmail>
Organisation: ITronic
An: DBMail mailinglist <dbmail [at] dbmail>

Thats the content of the mail I got.

I should NEVER see this line because this folder is owned
by a different user.

Attachments:

signature.asc (0.26 KB)

paul at nfg

Jul 23, 2013, 5:29 AM

Post #15 of 28 (213 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

On 07/23/2013 12:55 PM, Reindl Harald wrote:

> the "invalid sequence" is mostly triggerable by "empty trash"
> with https://addons.mozilla.org/en-us/thunderbird/addon/xpunge/
> if you have it configured to expunge any folders and empty all
> trashes

I've been seeing those myself. I think I know what causes that. I'll
update the unit-tests, and do a fix.

> what scares me currently more is this below
>
> if users get contents of other users this is major critical
> until now i never seens this, but i hestitate currently
> to consider a upgrade on the production machines

From what I understand, it looks like somewhere stack memory isn't
cleaned properly before usage.

I have never seen this myself, but I'll install Opera Mail on a windows
machine and see if I can reproduce it.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Jul 23, 2013, 7:55 AM

Post #16 of 28 (210 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 23.07.2013 14:29, schrieb Paul J Stevens:
> On 07/23/2013 12:55 PM, Reindl Harald wrote:
>
>> the "invalid sequence" is mostly triggerable by "empty trash"
>> with https://addons.mozilla.org/en-us/thunderbird/addon/xpunge/
>> if you have it configured to expunge any folders and empty all
>> trashes
>
> I've been seeing those myself. I think I know what causes that. I'll
> update the unit-tests, and do a fix.

thank you

this seems to be serious, i am about moving 1000 test messages
between folders and if one started to respond with invalid
sequence it does no longer accept move messages there until
restart thunderbird

i look if i get some useful from wireshark...........

>> what scares me currently more is this below
>>
>> if users get contents of other users this is major critical
>> until now i never seens this, but i hestitate currently
>> to consider a upgrade on the production machines
>
> From what I understand, it looks like somewhere stack memory isn't
> cleaned properly before usage.
>
> I have never seen this myself, but I'll install Opera Mail on a windows
> machine and see if I can reproduce it

hopefully we can track this down - if users see mails from
other customers i need a one-way-ticket to a country in
the south :-)

Attachments:

signature.asc (0.26 KB)

h.reindl at thelounge

Jul 23, 2013, 8:10 AM

Post #17 of 28 (211 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 23.07.2013 16:55, schrieb Reindl Harald:
> Am 23.07.2013 14:29, schrieb Paul J Stevens:
>> On 07/23/2013 12:55 PM, Reindl Harald wrote:
>>> the "invalid sequence" is mostly triggerable by "empty trash"
>>> with https://addons.mozilla.org/en-us/thunderbird/addon/xpunge/
>>> if you have it configured to expunge any folders and empty all
>>> trashes
>>
>> I've been seeing those myself. I think I know what causes that. I'll
>> update the unit-tests, and do a fix.
>
> thank you
>
> this seems to be serious, i am about moving 1000 test messages
> between folders and if one started to respond with invalid
> sequence it does no longer accept move messages there until
> restart thunderbird
>
> i look if i get some useful from wireshark...........

attached wireshark while switch between two folders
responding with "invalid sequence"

Attachments:

wireshark.gif (46.9 KB)

signature.asc (0.26 KB)

paul at nfg

Jul 23, 2013, 11:12 AM

Post #18 of 28 (210 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

On 07/23/2013 05:10 PM, Reindl Harald wrote:

> attached wireshark while switch between two folders
> responding with "invalid sequence"

Wireshark screenshots are not very useful. pcap files are.

Still, looking at the screenshot: you must disable TLS on the
connection, or wireshark can't decode the SSL jabber.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Jul 24, 2013, 5:26 AM

Post #19 of 28 (201 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 23.07.2013 20:12, schrieb Paul J Stevens:
> On 07/23/2013 05:10 PM, Reindl Harald wrote:
>
>> attached wireshark while switch between two folders
>> responding with "invalid sequence"
>
>
> Wireshark screenshots are not very useful. pcap files are.
>
> Still, looking at the screenshot: you must disable TLS on the
> connection, or wireshark can't decode the SSL jabber

well, attached a debuglog

to reproduce:

* Thunderbird
* create 3 folders (Test 1, Test 2, Test 3)
* put 1000 generated messages in one of them
* drag&drop the 1000 messages between this folders around
* finally if you select the in the meantime empty ones -> "invalid sequence"

this is pretty new, the same normally only happens by "empty trash"
with the xpunge-extension while trash is still empty since years
and the new incarnation happens also in Outlook with a unspecific
error "command failed"

Attachments:

debuglog.txt (70.1 KB)

signature.asc (0.26 KB)

paul at nfg

Jul 24, 2013, 7:23 AM

Post #20 of 28 (200 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

On 24-07-13 14:26, Reindl Harald wrote:
>
>
> Am 23.07.2013 20:12, schrieb Paul J Stevens:
>> On 07/23/2013 05:10 PM, Reindl Harald wrote:
>>
>>> attached wireshark while switch between two folders responding
>>> with "invalid sequence"
>>
>>
>> Wireshark screenshots are not very useful. pcap files are.
>>
>> Still, looking at the screenshot: you must disable TLS on the
>> connection, or wireshark can't decode the SSL jabber
>
> well, attached a debuglog
>
> to reproduce:
>
> * Thunderbird * create 3 folders (Test 1, Test 2, Test 3) * put
> 1000 generated messages in one of them * drag&drop the 1000
> messages between this folders around * finally if you select the in
> the meantime empty ones -> "invalid sequence"

fixed now.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Jul 24, 2013, 8:08 AM

Post #21 of 28 (200 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 24.07.2013 16:23, schrieb Paul J Stevens:
> On 24-07-13 14:26, Reindl Harald wrote:
>> to reproduce:
>>
>> Thunderbird * create 3 folders (Test 1, Test 2, Test 3) * put
>> 1000 generated messages in one of them * drag&drop the 1000
>> messages between this folders around * finally if you select the in
>> the meantime empty ones -> "invalid sequence"
>
> fixed now

*nearly* perfect

i moved the 1000 messages a few times around between the 3 folders
thunderbird-filters are set to take 100 and move them in local
folders 0-9 and due this another "invalid sequence"

the testcase i subject "Testmail 0001" - "Testmail 0999" and 10 filters
"starts with Testmail 0", "starts with Testmail 1" to localdir 0-9

(large) debuglog attached (tar.xz for best compression)

line 384415, i did not strip the rest so you may have a bigger picture
_______________________________

IMHO we should now only do fine-debugging and heavy testing
on several clients without any other changes in HEAD and
finally release a 3.1.1 which could become the best release
in the dbmail-history at all

@Harald Leithner: please join this party :-)

Attachments:

debuglog.tar.xz (423 KB)

signature.asc (0.26 KB)

harald.leithner at itronic

Jul 24, 2013, 8:20 AM

Post #22 of 28 (200 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 24.07.2013, 17:08 Uhr, schrieb Reindl Harald <h.reindl [at] thelounge>:

> Am 24.07.2013 16:23, schrieb Paul J Stevens:
>> On 24-07-13 14:26, Reindl Harald wrote:
>>> to reproduce:
>>>
>>> Thunderbird * create 3 folders (Test 1, Test 2, Test 3) * put
>>> 1000 generated messages in one of them * drag&drop the 1000
>>> messages between this folders around * finally if you select the in
>>> the meantime empty ones -> "invalid sequence"
>>
>> fixed now
>
> *nearly* perfect
>
> i moved the 1000 messages a few times around between the 3 folders
> thunderbird-filters are set to take 100 and move them in local
> folders 0-9 and due this another "invalid sequence"
>
> the testcase i subject "Testmail 0001" - "Testmail 0999" and 10 filters
> "starts with Testmail 0", "starts with Testmail 1" to localdir 0-9
>
> (large) debuglog attached (tar.xz for best compression)
>
> line 384415, i did not strip the rest so you may have a bigger picture
> _______________________________
>
> IMHO we should now only do fine-debugging and heavy testing
> on several clients without any other changes in HEAD and
> finally release a 3.1.1 which could become the best release
> in the dbmail-history at all
>
> @Harald Leithner: please join this party :-)

I don't have problems with thunderbird ;-) But I ask the client if he
still have problems with delayed E-Mails, I will post the replay to the
list.

Also I like patch levels, after fixing this problem a 3.1.1 would be great
maybe with a branch for 3.2.0 and 3.1.2?

--
Harald Leithner

ITronic
Vogelweidplatz 12, 1150 Wien, Austria
Tel: +43-1-786 23 88
Fax: +43-1-98 52 077
Mobil: +43-699-123 78 4 78
Mail: leithner [at] itronic | itronic.at
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Jul 24, 2013, 8:24 AM

Post #23 of 28 (200 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 24.07.2013 17:20, schrieb Harald Leithner:
>> Am 24.07.2013, 17:08 Uhr, schrieb Reindl Harald <h.reindl [at] thelounge>:
>> IMHO we should now only do fine-debugging and heavy testing
>> on several clients without any other changes in HEAD and
>> finally release a 3.1.1 which could become the best release
>> in the dbmail-history at all
>>
>> @Harald Leithner: please join this party :-)
>
> I don't have problems with thunderbird ;-) But I ask the client if he still have problems with delayed E-Mails, I
> will post the replay to the list.

i meant a test-party for all dbmail-components at whatever client combinations :-)

next monday i migrate our infrastructure to PHP 5.4
next thursday/friday i planned to do a dist-upgrade on around 20 servers
well, dbmail 3.1.x rollout between this fits in my timeframes :-)

> Also I like patch levels, after fixing this problem a 3.1.1 would be great maybe with a branch for 3.2.0 and 3.1.2?

me too

Attachments:

signature.asc (0.26 KB)

paul at nfg

Jul 24, 2013, 12:16 PM

Post #24 of 28 (197 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

On 07/24/2013 05:08 PM, Reindl Harald wrote:

> *nearly* perfect
>
> i moved the 1000 messages a few times around between the 3 folders
> thunderbird-filters are set to take 100 and move them in local
> folders 0-9 and due this another "invalid sequence"

Mmm, looking at the logs, I seriously doubt that is a regression. Looks
to me like a synchronisation problem.

C < [52:89 uid store 136001:136986 +FLAGS (\Deleted \Seen)
C < [12:90 expunge
C < [354:91 uid store
136013:136014,136055:136056,136094:136095,136145:136146,136174:136175,136205:136206,136248:136249,136263:136264,136316:136317,136345:136346,136377:136378,136393:136394,136434:136435,136464:136465,136493:136494,136703:136704,136754:136755,136793:136794,136847:136848,136874:136875,136906:136907,136934:136935,136985:136986
+Flags (\Deleted)

So: the client stores the \deleted flag on 986 messages, and issues an
expunge. Before all the expunge results are received thunderbird tries
to store the \deleted flag again on part of the list. But that list has
already been 'cleared' by dbmail, leading to a BAD response. So: the
list should only be cleared *after* the expunge command finishes.

Tricky, I expect.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin

* Premium Hosting Services and Web Application Consultancy *

www.nfg.nl/info [at] nfg/+31.85.877.99.97
________________________________________________________________
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

h.reindl at thelounge

Jul 24, 2013, 12:37 PM

Post #25 of 28 (197 views)
Permalink

Re: Outlook 2013 Imap implementation is broken [In reply to]

Am 24.07.2013 21:16, schrieb Paul J Stevens:
> On 07/24/2013 05:08 PM, Reindl Harald wrote:
>> *nearly* perfect
>>
>> i moved the 1000 messages a few times around between the 3 folders
>> thunderbird-filters are set to take 100 and move them in local
>> folders 0-9 and due this another "invalid sequence"
>
>
> Mmm, looking at the logs, I seriously doubt that is a regression. Looks
> to me like a synchronisation problem.

i also doubt it is a regression, i see the "invalid sequence"
since a longer time randomly appear, it seems to be far better
with the last patch, but i thought maybe we can finally
eliminate it now

> C < [52:89 uid store 136001:136986 +FLAGS (\Deleted \Seen)
> C < [12:90 expunge
> C < [354:91 uid store
> 136013:136014,136055:136056,136094:136095,136145:136146,136174:136175,136205:136206,136248:136249,136263:136264,136316:136317,136345:136346,136377:136378,136393:136394,136434:136435,136464:136465,136493:136494,136703:136704,136754:136755,136793:136794,136847:136848,136874:136875,136906:136907,136934:136935,136985:136986
> +Flags (\Deleted)
>
> So: the client stores the \deleted flag on 986 messages, and issues an
> expunge. Before all the expunge results are received thunderbird tries
> to store the \deleted flag again on part of the list. But that list has
> already been 'cleared' by dbmail, leading to a BAD response. So: the
> list should only be cleared *after* the expunge command finishes.
>
> Tricky, I expect

i can reproduce this as often s i want with this 1000 messages
and the filter, i simply put them back to one of the IMAP folders
move them around there and trigger the filter again

as i thought "hey, ivalid sequence is gone" dbmail was not in
debug-mode as it appeared at the end of the filter messages and
it was predictable to reproduce

however, i can not say how often this happen under normal load
until i finshed a lot of testing and rollout 3.1.x because it's
hard to reproduce the real life workload in a syntetic test

Attachments:

signature.asc (0.26 KB)

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads