Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DBMail: dev

[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages

 

 

DBMail dev RSS feed   Index | Next | Previous | View Threaded


bugtrack at dbmail

Dec 21, 2011, 9:58 AM

Post #1 of 10 (631 views)
Permalink
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages

A NOTE has been added to this issue.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 21-Dec-11 18:58 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.

Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.

Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.

I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix

Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk


======================================================================

----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)

I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.

For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.

Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...

Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
======================================================================

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


bugtrack at dbmail

Dec 22, 2011, 12:27 AM

Post #2 of 10 (596 views)
Permalink
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages [In reply to]

A NOTE has been added to this issue.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 22-Dec-11 09:27 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.

Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.

Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.

I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix

Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk


======================================================================

----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)

I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.

For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.

Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...

----------------------------------------------------------------------
(0003372) paul (administrator) - 22-Dec-11 09:27
http://dbmail.org/mantis/view.php?id=924#c3372
----------------------------------------------------------------------
Without logs there's precious little I can do.

dbmail-util -yM will migrate all physmessage to 3.0 style storage
dbmail-util -ym will migrate 10000 physmessages
dbmail-util -ym 100 will migrate 100 physmessages

note however, that migrating physmessages is *not* required.

in all cases

dbmail-util -by

is required since the migration script drops and creates the headercache
tables that need to be re-filled.

So please include logs that show SQL (level 511)

Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
22-Dec-11 09:27 paul Note Added: 0003372
======================================================================

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


bugtrack at dbmail

Dec 22, 2011, 1:07 PM

Post #3 of 10 (598 views)
Permalink
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages [In reply to]

A NOTE has been added to this issue.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 22-Dec-11 22:07 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.

Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.

Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.

I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix

Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk


======================================================================

----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)

I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.

For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.

Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...

----------------------------------------------------------------------
(0003372) paul (administrator) - 22-Dec-11 09:27
http://dbmail.org/mantis/view.php?id=924#c3372
----------------------------------------------------------------------
Without logs there's precious little I can do.

dbmail-util -yM will migrate all physmessage to 3.0 style storage
dbmail-util -ym will migrate 10000 physmessages
dbmail-util -ym 100 will migrate 100 physmessages

note however, that migrating physmessages is *not* required.

in all cases

dbmail-util -by

is required since the migration script drops and creates the headercache
tables that need to be re-filled.

So please include logs that show SQL (level 511)

----------------------------------------------------------------------
(0003373) mverbert (reporter) - 22-Dec-11 22:07
http://dbmail.org/mantis/view.php?id=924#c3373
----------------------------------------------------------------------
OK, I ran the experiment once more with the correct log level (511).

The log files are the following (in order of generation):
1) migration.log: output of "dbmail -yM"
2) migration_rest.log: output of "dbmail -yM -m 50000" ; most probably not
very interesting. At the end of the process "dbmail -M" shows no more
messages to migrate.
3) uncache.log: output of "dbmail -by"
4) after_uncache.log: output of "dbmail -b" showing that no progress was
made

I restore the db and tried just "dbmail -by" generating uncache2.log.
Needless to say that it didn't work, but that was just for the sake of
experimenting without migrating the physmessages.

The log files are well above the 150k limit of Mantis, so I put them
there:
http://moutarde.dyndns.org/dbmail/

Let me know if you need more information.
Thanks for your support and Merry Christmas !

Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
22-Dec-11 09:27 paul Note Added: 0003372
22-Dec-11 22:07 mverbert Note Added: 0003373
======================================================================

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


bugtrack at dbmail

Dec 22, 2011, 2:07 PM

Post #4 of 10 (596 views)
Permalink
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages [In reply to]

A NOTE has been added to this issue.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 22-Dec-11 23:07 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.

Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.

Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.

I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix

Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk


======================================================================

----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)

I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.

For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.

Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...

----------------------------------------------------------------------
(0003372) paul (administrator) - 22-Dec-11 09:27
http://dbmail.org/mantis/view.php?id=924#c3372
----------------------------------------------------------------------
Without logs there's precious little I can do.

dbmail-util -yM will migrate all physmessage to 3.0 style storage
dbmail-util -ym will migrate 10000 physmessages
dbmail-util -ym 100 will migrate 100 physmessages

note however, that migrating physmessages is *not* required.

in all cases

dbmail-util -by

is required since the migration script drops and creates the headercache
tables that need to be re-filled.

So please include logs that show SQL (level 511)

----------------------------------------------------------------------
(0003373) mverbert (reporter) - 22-Dec-11 22:07
http://dbmail.org/mantis/view.php?id=924#c3373
----------------------------------------------------------------------
OK, I ran the experiment once more with the correct log level (511).

The log files are the following (in order of generation):
1) migration.log: output of "dbmail -yM"
2) migration_rest.log: output of "dbmail -yM -m 50000" ; most probably not
very interesting. At the end of the process "dbmail -M" shows no more
messages to migrate.
3) uncache.log: output of "dbmail -by"
4) after_uncache.log: output of "dbmail -b" showing that no progress was
made

I restore the db and tried just "dbmail -by" generating uncache2.log.
Needless to say that it didn't work, but that was just for the sake of
experimenting without migrating the physmessages.

The log files are well above the 150k limit of Mantis, so I put them
there:
http://moutarde.dyndns.org/dbmail/

Let me know if you need more information.
Thanks for your support and Merry Christmas !

----------------------------------------------------------------------
(0003374) paul (administrator) - 22-Dec-11 23:07
http://dbmail.org/mantis/view.php?id=924#c3374
----------------------------------------------------------------------
The uncache.log definitely doesn't look right. But I have never tested the
order in which you do stuff.

Normal steps to follow would be:

apply schema migration (required)
rebuild header cache (required)
migrate storage (optional)

So leave out the last one, and just run the first. Everything must work
then apart from imap-search on message bodies.

When it does, only then try the last step.

I'll try to reproduce your steps and see if fixing it isn't too difficult.

Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
22-Dec-11 09:27 paul Note Added: 0003372
22-Dec-11 22:07 mverbert Note Added: 0003373
22-Dec-11 23:07 paul Note Added: 0003374
======================================================================

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


bugtrack at dbmail

Dec 22, 2011, 10:34 PM

Post #5 of 10 (616 views)
Permalink
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages [In reply to]

A NOTE has been added to this issue.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 23-Dec-11 07:34 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.

Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.

Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.

I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix

Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk


======================================================================

----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)

I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.

For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.

Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...

----------------------------------------------------------------------
(0003372) paul (administrator) - 22-Dec-11 09:27
http://dbmail.org/mantis/view.php?id=924#c3372
----------------------------------------------------------------------
Without logs there's precious little I can do.

dbmail-util -yM will migrate all physmessage to 3.0 style storage
dbmail-util -ym will migrate 10000 physmessages
dbmail-util -ym 100 will migrate 100 physmessages

note however, that migrating physmessages is *not* required.

in all cases

dbmail-util -by

is required since the migration script drops and creates the headercache
tables that need to be re-filled.

So please include logs that show SQL (level 511)

----------------------------------------------------------------------
(0003373) mverbert (reporter) - 22-Dec-11 22:07
http://dbmail.org/mantis/view.php?id=924#c3373
----------------------------------------------------------------------
OK, I ran the experiment once more with the correct log level (511).

The log files are the following (in order of generation):
1) migration.log: output of "dbmail -yM"
2) migration_rest.log: output of "dbmail -yM -m 50000" ; most probably not
very interesting. At the end of the process "dbmail -M" shows no more
messages to migrate.
3) uncache.log: output of "dbmail -by"
4) after_uncache.log: output of "dbmail -b" showing that no progress was
made

I restore the db and tried just "dbmail -by" generating uncache2.log.
Needless to say that it didn't work, but that was just for the sake of
experimenting without migrating the physmessages.

The log files are well above the 150k limit of Mantis, so I put them
there:
http://moutarde.dyndns.org/dbmail/

Let me know if you need more information.
Thanks for your support and Merry Christmas !

----------------------------------------------------------------------
(0003374) paul (administrator) - 22-Dec-11 23:07
http://dbmail.org/mantis/view.php?id=924#c3374
----------------------------------------------------------------------
The uncache.log definitely doesn't look right. But I have never tested the
order in which you do stuff.

Normal steps to follow would be:

apply schema migration (required)
rebuild header cache (required)
migrate storage (optional)

So leave out the last one, and just run the first. Everything must work
then apart from imap-search on message bodies.

When it does, only then try the last step.

I'll try to reproduce your steps and see if fixing it isn't too difficult.

----------------------------------------------------------------------
(0003375) mverbert (reporter) - 23-Dec-11 07:34
http://dbmail.org/mantis/view.php?id=924#c3375
----------------------------------------------------------------------
It seems the last part of my previous message was not completely clear:
uncache2.log represents the "rebuild header cache" of the "normal steps",
given the database was fully restored from a backup (ie. including drop of
database) just before.
Thanks for your help.

Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
22-Dec-11 09:27 paul Note Added: 0003372
22-Dec-11 22:07 mverbert Note Added: 0003373
22-Dec-11 23:07 paul Note Added: 0003374
23-Dec-11 07:34 mverbert Note Added: 0003375
======================================================================

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


bugtrack at dbmail

Dec 23, 2011, 1:23 AM

Post #6 of 10 (600 views)
Permalink
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages [In reply to]

A NOTE has been added to this issue.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 23-Dec-11 10:23 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.

Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.

Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.

I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix

Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk


======================================================================

----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)

I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.

For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.

Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...

----------------------------------------------------------------------
(0003372) paul (administrator) - 22-Dec-11 09:27
http://dbmail.org/mantis/view.php?id=924#c3372
----------------------------------------------------------------------
Without logs there's precious little I can do.

dbmail-util -yM will migrate all physmessage to 3.0 style storage
dbmail-util -ym will migrate 10000 physmessages
dbmail-util -ym 100 will migrate 100 physmessages

note however, that migrating physmessages is *not* required.

in all cases

dbmail-util -by

is required since the migration script drops and creates the headercache
tables that need to be re-filled.

So please include logs that show SQL (level 511)

----------------------------------------------------------------------
(0003373) mverbert (reporter) - 22-Dec-11 22:07
http://dbmail.org/mantis/view.php?id=924#c3373
----------------------------------------------------------------------
OK, I ran the experiment once more with the correct log level (511).

The log files are the following (in order of generation):
1) migration.log: output of "dbmail -yM"
2) migration_rest.log: output of "dbmail -yM -m 50000" ; most probably not
very interesting. At the end of the process "dbmail -M" shows no more
messages to migrate.
3) uncache.log: output of "dbmail -by"
4) after_uncache.log: output of "dbmail -b" showing that no progress was
made

I restore the db and tried just "dbmail -by" generating uncache2.log.
Needless to say that it didn't work, but that was just for the sake of
experimenting without migrating the physmessages.

The log files are well above the 150k limit of Mantis, so I put them
there:
http://moutarde.dyndns.org/dbmail/

Let me know if you need more information.
Thanks for your support and Merry Christmas !

----------------------------------------------------------------------
(0003374) paul (administrator) - 22-Dec-11 23:07
http://dbmail.org/mantis/view.php?id=924#c3374
----------------------------------------------------------------------
The uncache.log definitely doesn't look right. But I have never tested the
order in which you do stuff.

Normal steps to follow would be:

apply schema migration (required)
rebuild header cache (required)
migrate storage (optional)

So leave out the last one, and just run the first. Everything must work
then apart from imap-search on message bodies.

When it does, only then try the last step.

I'll try to reproduce your steps and see if fixing it isn't too difficult.

----------------------------------------------------------------------
(0003375) mverbert (reporter) - 23-Dec-11 07:34
http://dbmail.org/mantis/view.php?id=924#c3375
----------------------------------------------------------------------
It seems the last part of my previous message was not completely clear:
uncache2.log represents the "rebuild header cache" of the "normal steps",
given the database was fully restored from a backup (ie. including drop of
database) just before.
Thanks for your help.

----------------------------------------------------------------------
(0003376) paul (administrator) - 23-Dec-11 10:23
http://dbmail.org/mantis/view.php?id=924#c3376
----------------------------------------------------------------------
Did some tests. Had an epiphany. You guys are running pg9. You *must* set
in postgresql.conf:

bytea_output = 'escape'

or else dbmail won't be able to interpret the query results involving
bytea fields.

Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
22-Dec-11 09:27 paul Note Added: 0003372
22-Dec-11 22:07 mverbert Note Added: 0003373
22-Dec-11 23:07 paul Note Added: 0003374
23-Dec-11 07:34 mverbert Note Added: 0003375
23-Dec-11 10:23 paul Note Added: 0003376
======================================================================

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


bugtrack at dbmail

Dec 23, 2011, 1:24 AM

Post #7 of 10 (599 views)
Permalink
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages [In reply to]

A NOTE has been added to this issue.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 23-Dec-11 10:24 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.

Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.

Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.

I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix

Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk


======================================================================

----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)

I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.

For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.

Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...

----------------------------------------------------------------------
(0003372) paul (administrator) - 22-Dec-11 09:27
http://dbmail.org/mantis/view.php?id=924#c3372
----------------------------------------------------------------------
Without logs there's precious little I can do.

dbmail-util -yM will migrate all physmessage to 3.0 style storage
dbmail-util -ym will migrate 10000 physmessages
dbmail-util -ym 100 will migrate 100 physmessages

note however, that migrating physmessages is *not* required.

in all cases

dbmail-util -by

is required since the migration script drops and creates the headercache
tables that need to be re-filled.

So please include logs that show SQL (level 511)

----------------------------------------------------------------------
(0003373) mverbert (reporter) - 22-Dec-11 22:07
http://dbmail.org/mantis/view.php?id=924#c3373
----------------------------------------------------------------------
OK, I ran the experiment once more with the correct log level (511).

The log files are the following (in order of generation):
1) migration.log: output of "dbmail -yM"
2) migration_rest.log: output of "dbmail -yM -m 50000" ; most probably not
very interesting. At the end of the process "dbmail -M" shows no more
messages to migrate.
3) uncache.log: output of "dbmail -by"
4) after_uncache.log: output of "dbmail -b" showing that no progress was
made

I restore the db and tried just "dbmail -by" generating uncache2.log.
Needless to say that it didn't work, but that was just for the sake of
experimenting without migrating the physmessages.

The log files are well above the 150k limit of Mantis, so I put them
there:
http://moutarde.dyndns.org/dbmail/

Let me know if you need more information.
Thanks for your support and Merry Christmas !

----------------------------------------------------------------------
(0003374) paul (administrator) - 22-Dec-11 23:07
http://dbmail.org/mantis/view.php?id=924#c3374
----------------------------------------------------------------------
The uncache.log definitely doesn't look right. But I have never tested the
order in which you do stuff.

Normal steps to follow would be:

apply schema migration (required)
rebuild header cache (required)
migrate storage (optional)

So leave out the last one, and just run the first. Everything must work
then apart from imap-search on message bodies.

When it does, only then try the last step.

I'll try to reproduce your steps and see if fixing it isn't too difficult.

----------------------------------------------------------------------
(0003375) mverbert (reporter) - 23-Dec-11 07:34
http://dbmail.org/mantis/view.php?id=924#c3375
----------------------------------------------------------------------
It seems the last part of my previous message was not completely clear:
uncache2.log represents the "rebuild header cache" of the "normal steps",
given the database was fully restored from a backup (ie. including drop of
database) just before.
Thanks for your help.

----------------------------------------------------------------------
(0003376) paul (administrator) - 23-Dec-11 10:23
http://dbmail.org/mantis/view.php?id=924#c3376
----------------------------------------------------------------------
Did some tests. Had an epiphany. You guys are running pg9. You *must* set
in postgresql.conf:

bytea_output = 'escape'

or else dbmail won't be able to interpret the query results involving
bytea fields.

----------------------------------------------------------------------
(0003377) paul (administrator) - 23-Dec-11 10:24
http://dbmail.org/mantis/view.php?id=924#c3377
----------------------------------------------------------------------
For your information, you can test by running one of these:

SELECT b.messageblk, b.is_header, TO_CHAR(p.internal_date, 'YYYY-MM-DD
HH24:MI:SS' ) FROM dbmail_messageblks b JOIN dbmail_physmessage p ON
b.physmessage_id=p.id WHERE b.physmessage_id = 56389 AND b.is_header = '1'

before migrating to the new-style storage and make sure the result looks
like a valid email message

Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
22-Dec-11 09:27 paul Note Added: 0003372
22-Dec-11 22:07 mverbert Note Added: 0003373
22-Dec-11 23:07 paul Note Added: 0003374
23-Dec-11 07:34 mverbert Note Added: 0003375
23-Dec-11 10:23 paul Note Added: 0003376
23-Dec-11 10:24 paul Note Added: 0003377
======================================================================

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


bugtrack at dbmail

Dec 25, 2011, 3:35 AM

Post #8 of 10 (585 views)
Permalink
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages [In reply to]

A NOTE has been added to this issue.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 25-Dec-11 12:35 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.

Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.

Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.

I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix

Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk


======================================================================

----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)

I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.

For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.

Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...

----------------------------------------------------------------------
(0003372) paul (administrator) - 22-Dec-11 09:27
http://dbmail.org/mantis/view.php?id=924#c3372
----------------------------------------------------------------------
Without logs there's precious little I can do.

dbmail-util -yM will migrate all physmessage to 3.0 style storage
dbmail-util -ym will migrate 10000 physmessages
dbmail-util -ym 100 will migrate 100 physmessages

note however, that migrating physmessages is *not* required.

in all cases

dbmail-util -by

is required since the migration script drops and creates the headercache
tables that need to be re-filled.

So please include logs that show SQL (level 511)

----------------------------------------------------------------------
(0003373) mverbert (reporter) - 22-Dec-11 22:07
http://dbmail.org/mantis/view.php?id=924#c3373
----------------------------------------------------------------------
OK, I ran the experiment once more with the correct log level (511).

The log files are the following (in order of generation):
1) migration.log: output of "dbmail -yM"
2) migration_rest.log: output of "dbmail -yM -m 50000" ; most probably not
very interesting. At the end of the process "dbmail -M" shows no more
messages to migrate.
3) uncache.log: output of "dbmail -by"
4) after_uncache.log: output of "dbmail -b" showing that no progress was
made

I restore the db and tried just "dbmail -by" generating uncache2.log.
Needless to say that it didn't work, but that was just for the sake of
experimenting without migrating the physmessages.

The log files are well above the 150k limit of Mantis, so I put them
there:
http://moutarde.dyndns.org/dbmail/

Let me know if you need more information.
Thanks for your support and Merry Christmas !

----------------------------------------------------------------------
(0003374) paul (administrator) - 22-Dec-11 23:07
http://dbmail.org/mantis/view.php?id=924#c3374
----------------------------------------------------------------------
The uncache.log definitely doesn't look right. But I have never tested the
order in which you do stuff.

Normal steps to follow would be:

apply schema migration (required)
rebuild header cache (required)
migrate storage (optional)

So leave out the last one, and just run the first. Everything must work
then apart from imap-search on message bodies.

When it does, only then try the last step.

I'll try to reproduce your steps and see if fixing it isn't too difficult.

----------------------------------------------------------------------
(0003375) mverbert (reporter) - 23-Dec-11 07:34
http://dbmail.org/mantis/view.php?id=924#c3375
----------------------------------------------------------------------
It seems the last part of my previous message was not completely clear:
uncache2.log represents the "rebuild header cache" of the "normal steps",
given the database was fully restored from a backup (ie. including drop of
database) just before.
Thanks for your help.

----------------------------------------------------------------------
(0003376) paul (administrator) - 23-Dec-11 10:23
http://dbmail.org/mantis/view.php?id=924#c3376
----------------------------------------------------------------------
Did some tests. Had an epiphany. You guys are running pg9. You *must* set
in postgresql.conf:

bytea_output = 'escape'

or else dbmail won't be able to interpret the query results involving
bytea fields.

----------------------------------------------------------------------
(0003377) paul (administrator) - 23-Dec-11 10:24
http://dbmail.org/mantis/view.php?id=924#c3377
----------------------------------------------------------------------
For your information, you can test by running one of these:

SELECT b.messageblk, b.is_header, TO_CHAR(p.internal_date, 'YYYY-MM-DD
HH24:MI:SS' ) FROM dbmail_messageblks b JOIN dbmail_physmessage p ON
b.physmessage_id=p.id WHERE b.physmessage_id = 56389 AND b.is_header = '1'

before migrating to the new-style storage and make sure the result looks
like a valid email message

----------------------------------------------------------------------
(0003378) mverbert (reporter) - 25-Dec-11 12:35
http://dbmail.org/mantis/view.php?id=924#c3378
----------------------------------------------------------------------
I did run the command mentioned in issue
http://dbmail.org/mantis/view.php?id=953 (ALTER DATABASE dbmail SET
bytea_output='escape';), and then everything started to work like a charm.
I believe you can close this issue if the original reporter agrees.
Thanks (and Merry Christmas)

Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
22-Dec-11 09:27 paul Note Added: 0003372
22-Dec-11 22:07 mverbert Note Added: 0003373
22-Dec-11 23:07 paul Note Added: 0003374
23-Dec-11 07:34 mverbert Note Added: 0003375
23-Dec-11 10:23 paul Note Added: 0003376
23-Dec-11 10:24 paul Note Added: 0003377
25-Dec-11 12:35 mverbert Note Added: 0003378
======================================================================

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


bugtrack at dbmail

Dec 26, 2011, 12:57 PM

Post #9 of 10 (574 views)
Permalink
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages [In reply to]

A NOTE has been added to this issue.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 26-Dec-11 21:57 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.

Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.

Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.

I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix

Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk


======================================================================

----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)

I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.

For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.

Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...

----------------------------------------------------------------------
(0003372) paul (administrator) - 22-Dec-11 09:27
http://dbmail.org/mantis/view.php?id=924#c3372
----------------------------------------------------------------------
Without logs there's precious little I can do.

dbmail-util -yM will migrate all physmessage to 3.0 style storage
dbmail-util -ym will migrate 10000 physmessages
dbmail-util -ym 100 will migrate 100 physmessages

note however, that migrating physmessages is *not* required.

in all cases

dbmail-util -by

is required since the migration script drops and creates the headercache
tables that need to be re-filled.

So please include logs that show SQL (level 511)

----------------------------------------------------------------------
(0003373) mverbert (reporter) - 22-Dec-11 22:07
http://dbmail.org/mantis/view.php?id=924#c3373
----------------------------------------------------------------------
OK, I ran the experiment once more with the correct log level (511).

The log files are the following (in order of generation):
1) migration.log: output of "dbmail -yM"
2) migration_rest.log: output of "dbmail -yM -m 50000" ; most probably not
very interesting. At the end of the process "dbmail -M" shows no more
messages to migrate.
3) uncache.log: output of "dbmail -by"
4) after_uncache.log: output of "dbmail -b" showing that no progress was
made

I restore the db and tried just "dbmail -by" generating uncache2.log.
Needless to say that it didn't work, but that was just for the sake of
experimenting without migrating the physmessages.

The log files are well above the 150k limit of Mantis, so I put them
there:
http://moutarde.dyndns.org/dbmail/

Let me know if you need more information.
Thanks for your support and Merry Christmas !

----------------------------------------------------------------------
(0003374) paul (administrator) - 22-Dec-11 23:07
http://dbmail.org/mantis/view.php?id=924#c3374
----------------------------------------------------------------------
The uncache.log definitely doesn't look right. But I have never tested the
order in which you do stuff.

Normal steps to follow would be:

apply schema migration (required)
rebuild header cache (required)
migrate storage (optional)

So leave out the last one, and just run the first. Everything must work
then apart from imap-search on message bodies.

When it does, only then try the last step.

I'll try to reproduce your steps and see if fixing it isn't too difficult.

----------------------------------------------------------------------
(0003375) mverbert (reporter) - 23-Dec-11 07:34
http://dbmail.org/mantis/view.php?id=924#c3375
----------------------------------------------------------------------
It seems the last part of my previous message was not completely clear:
uncache2.log represents the "rebuild header cache" of the "normal steps",
given the database was fully restored from a backup (ie. including drop of
database) just before.
Thanks for your help.

----------------------------------------------------------------------
(0003376) paul (administrator) - 23-Dec-11 10:23
http://dbmail.org/mantis/view.php?id=924#c3376
----------------------------------------------------------------------
Did some tests. Had an epiphany. You guys are running pg9. You *must* set
in postgresql.conf:

bytea_output = 'escape'

or else dbmail won't be able to interpret the query results involving
bytea fields.

----------------------------------------------------------------------
(0003377) paul (administrator) - 23-Dec-11 10:24
http://dbmail.org/mantis/view.php?id=924#c3377
----------------------------------------------------------------------
For your information, you can test by running one of these:

SELECT b.messageblk, b.is_header, TO_CHAR(p.internal_date, 'YYYY-MM-DD
HH24:MI:SS' ) FROM dbmail_messageblks b JOIN dbmail_physmessage p ON
b.physmessage_id=p.id WHERE b.physmessage_id = 56389 AND b.is_header = '1'

before migrating to the new-style storage and make sure the result looks
like a valid email message

----------------------------------------------------------------------
(0003378) mverbert (reporter) - 25-Dec-11 12:35
http://dbmail.org/mantis/view.php?id=924#c3378
----------------------------------------------------------------------
I did run the command mentioned in issue
http://dbmail.org/mantis/view.php?id=953 (ALTER DATABASE dbmail SET
bytea_output='escape';), and then everything started to work like a charm.
I believe you can close this issue if the original reporter agrees.
Thanks (and Merry Christmas)

----------------------------------------------------------------------
(0003379) gerritcap (reporter) - 26-Dec-11 21:57
http://dbmail.org/mantis/view.php?id=924#c3379
----------------------------------------------------------------------
I haven't been able to check the solution yet, but I assume, as I was also
using postgres 9 that this was the problem, trying to find the time to
double check...

Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
22-Dec-11 09:27 paul Note Added: 0003372
22-Dec-11 22:07 mverbert Note Added: 0003373
22-Dec-11 23:07 paul Note Added: 0003374
23-Dec-11 07:34 mverbert Note Added: 0003375
23-Dec-11 10:23 paul Note Added: 0003376
23-Dec-11 10:24 paul Note Added: 0003377
25-Dec-11 12:35 mverbert Note Added: 0003378
26-Dec-11 21:57 gerritcap Note Added: 0003379
======================================================================

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


bugtrack at dbmail

Jan 1, 2012, 3:21 AM

Post #10 of 10 (542 views)
Permalink
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages [In reply to]

The following issue has been RESOLVED.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: resolved
target:
Resolution: no change required
Fixed in Version:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 01-Jan-12 12:21 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.

Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.

Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.

I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix

Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk


======================================================================

----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)

I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.

For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.

Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...

----------------------------------------------------------------------
(0003372) paul (administrator) - 22-Dec-11 09:27
http://dbmail.org/mantis/view.php?id=924#c3372
----------------------------------------------------------------------
Without logs there's precious little I can do.

dbmail-util -yM will migrate all physmessage to 3.0 style storage
dbmail-util -ym will migrate 10000 physmessages
dbmail-util -ym 100 will migrate 100 physmessages

note however, that migrating physmessages is *not* required.

in all cases

dbmail-util -by

is required since the migration script drops and creates the headercache
tables that need to be re-filled.

So please include logs that show SQL (level 511)

----------------------------------------------------------------------
(0003373) mverbert (reporter) - 22-Dec-11 22:07
http://dbmail.org/mantis/view.php?id=924#c3373
----------------------------------------------------------------------
OK, I ran the experiment once more with the correct log level (511).

The log files are the following (in order of generation):
1) migration.log: output of "dbmail -yM"
2) migration_rest.log: output of "dbmail -yM -m 50000" ; most probably not
very interesting. At the end of the process "dbmail -M" shows no more
messages to migrate.
3) uncache.log: output of "dbmail -by"
4) after_uncache.log: output of "dbmail -b" showing that no progress was
made

I restore the db and tried just "dbmail -by" generating uncache2.log.
Needless to say that it didn't work, but that was just for the sake of
experimenting without migrating the physmessages.

The log files are well above the 150k limit of Mantis, so I put them
there:
http://moutarde.dyndns.org/dbmail/

Let me know if you need more information.
Thanks for your support and Merry Christmas !

----------------------------------------------------------------------
(0003374) paul (administrator) - 22-Dec-11 23:07
http://dbmail.org/mantis/view.php?id=924#c3374
----------------------------------------------------------------------
The uncache.log definitely doesn't look right. But I have never tested the
order in which you do stuff.

Normal steps to follow would be:

apply schema migration (required)
rebuild header cache (required)
migrate storage (optional)

So leave out the last one, and just run the first. Everything must work
then apart from imap-search on message bodies.

When it does, only then try the last step.

I'll try to reproduce your steps and see if fixing it isn't too difficult.

----------------------------------------------------------------------
(0003375) mverbert (reporter) - 23-Dec-11 07:34
http://dbmail.org/mantis/view.php?id=924#c3375
----------------------------------------------------------------------
It seems the last part of my previous message was not completely clear:
uncache2.log represents the "rebuild header cache" of the "normal steps",
given the database was fully restored from a backup (ie. including drop of
database) just before.
Thanks for your help.

----------------------------------------------------------------------
(0003376) paul (administrator) - 23-Dec-11 10:23
http://dbmail.org/mantis/view.php?id=924#c3376
----------------------------------------------------------------------
Did some tests. Had an epiphany. You guys are running pg9. You *must* set
in postgresql.conf:

bytea_output = 'escape'

or else dbmail won't be able to interpret the query results involving
bytea fields.

----------------------------------------------------------------------
(0003377) paul (administrator) - 23-Dec-11 10:24
http://dbmail.org/mantis/view.php?id=924#c3377
----------------------------------------------------------------------
For your information, you can test by running one of these:

SELECT b.messageblk, b.is_header, TO_CHAR(p.internal_date, 'YYYY-MM-DD
HH24:MI:SS' ) FROM dbmail_messageblks b JOIN dbmail_physmessage p ON
b.physmessage_id=p.id WHERE b.physmessage_id = 56389 AND b.is_header = '1'

before migrating to the new-style storage and make sure the result looks
like a valid email message

----------------------------------------------------------------------
(0003378) mverbert (reporter) - 25-Dec-11 12:35
http://dbmail.org/mantis/view.php?id=924#c3378
----------------------------------------------------------------------
I did run the command mentioned in issue
http://dbmail.org/mantis/view.php?id=953 (ALTER DATABASE dbmail SET
bytea_output='escape';), and then everything started to work like a charm.
I believe you can close this issue if the original reporter agrees.
Thanks (and Merry Christmas)

----------------------------------------------------------------------
(0003379) gerritcap (reporter) - 26-Dec-11 21:57
http://dbmail.org/mantis/view.php?id=924#c3379
----------------------------------------------------------------------
I haven't been able to check the solution yet, but I assume, as I was also
using postgres 9 that this was the problem, trying to find the time to
double check...

----------------------------------------------------------------------
(0003381) paul (administrator) - 01-Jan-12 12:21
http://dbmail.org/mantis/view.php?id=924#c3381
----------------------------------------------------------------------
I'm closing this report, since it's a configuration issue and not a bug.

Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
22-Dec-11 09:27 paul Note Added: 0003372
22-Dec-11 22:07 mverbert Note Added: 0003373
22-Dec-11 23:07 paul Note Added: 0003374
23-Dec-11 07:34 mverbert Note Added: 0003375
23-Dec-11 10:23 paul Note Added: 0003376
23-Dec-11 10:24 paul Note Added: 0003377
25-Dec-11 12:35 mverbert Note Added: 0003378
26-Dec-11 21:57 gerritcap Note Added: 0003379
01-Jan-12 12:21 paul Note Added: 0003381
01-Jan-12 12:21 paul Status new => resolved
01-Jan-12 12:21 paul Resolution open => no change
required
======================================================================

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev

DBMail dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.