Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DBMail: dev
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages
 

Index | Next | Previous | View Flat


bugtrack at dbmail

Dec 22, 2011, 10:34 PM


Views: 602
Permalink
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages [In reply to]

A NOTE has been added to this issue.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 23-Dec-11 07:34 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.

Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.

Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.

I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix

Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk


======================================================================

----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)

I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.

For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.

Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...

----------------------------------------------------------------------
(0003372) paul (administrator) - 22-Dec-11 09:27
http://dbmail.org/mantis/view.php?id=924#c3372
----------------------------------------------------------------------
Without logs there's precious little I can do.

dbmail-util -yM will migrate all physmessage to 3.0 style storage
dbmail-util -ym will migrate 10000 physmessages
dbmail-util -ym 100 will migrate 100 physmessages

note however, that migrating physmessages is *not* required.

in all cases

dbmail-util -by

is required since the migration script drops and creates the headercache
tables that need to be re-filled.

So please include logs that show SQL (level 511)

----------------------------------------------------------------------
(0003373) mverbert (reporter) - 22-Dec-11 22:07
http://dbmail.org/mantis/view.php?id=924#c3373
----------------------------------------------------------------------
OK, I ran the experiment once more with the correct log level (511).

The log files are the following (in order of generation):
1) migration.log: output of "dbmail -yM"
2) migration_rest.log: output of "dbmail -yM -m 50000" ; most probably not
very interesting. At the end of the process "dbmail -M" shows no more
messages to migrate.
3) uncache.log: output of "dbmail -by"
4) after_uncache.log: output of "dbmail -b" showing that no progress was
made

I restore the db and tried just "dbmail -by" generating uncache2.log.
Needless to say that it didn't work, but that was just for the sake of
experimenting without migrating the physmessages.

The log files are well above the 150k limit of Mantis, so I put them
there:
http://moutarde.dyndns.org/dbmail/

Let me know if you need more information.
Thanks for your support and Merry Christmas !

----------------------------------------------------------------------
(0003374) paul (administrator) - 22-Dec-11 23:07
http://dbmail.org/mantis/view.php?id=924#c3374
----------------------------------------------------------------------
The uncache.log definitely doesn't look right. But I have never tested the
order in which you do stuff.

Normal steps to follow would be:

apply schema migration (required)
rebuild header cache (required)
migrate storage (optional)

So leave out the last one, and just run the first. Everything must work
then apart from imap-search on message bodies.

When it does, only then try the last step.

I'll try to reproduce your steps and see if fixing it isn't too difficult.

----------------------------------------------------------------------
(0003375) mverbert (reporter) - 23-Dec-11 07:34
http://dbmail.org/mantis/view.php?id=924#c3375
----------------------------------------------------------------------
It seems the last part of my previous message was not completely clear:
uncache2.log represents the "rebuild header cache" of the "normal steps",
given the database was fully restored from a backup (ie. including drop of
database) just before.
Thanks for your help.

Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
22-Dec-11 09:27 paul Note Added: 0003372
22-Dec-11 22:07 mverbert Note Added: 0003373
22-Dec-11 23:07 paul Note Added: 0003374
23-Dec-11 07:34 mverbert Note Added: 0003375
======================================================================

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev

Subject User Time
[DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages bugtrack at dbmail Dec 21, 2011, 9:58 AM
    [DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages bugtrack at dbmail Dec 22, 2011, 12:27 AM
    [DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages bugtrack at dbmail Dec 22, 2011, 1:07 PM
    [DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages bugtrack at dbmail Dec 22, 2011, 2:07 PM
    [DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages bugtrack at dbmail Dec 22, 2011, 10:34 PM
    [DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages bugtrack at dbmail Dec 23, 2011, 1:23 AM
    [DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages bugtrack at dbmail Dec 23, 2011, 1:24 AM
    [DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages bugtrack at dbmail Dec 25, 2011, 3:35 AM
    [DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages bugtrack at dbmail Dec 26, 2011, 12:57 PM
    [DBMail 0000924]: dbmail-util -by doesn't fix un-cached physmessages bugtrack at dbmail Jan 1, 2012, 3:21 AM

  Index | Next | Previous | View Flat
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.