Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DBMail: dev

dbmail-util not cleaning messages

 

 

DBMail dev RSS feed   Index | Next | Previous | View Threaded


mysql.jorge at decimal

Sep 19, 2009, 11:07 AM

Post #1 of 9 (1742 views)
Permalink
dbmail-util not cleaning messages

Paul/Jon,



I ask your help on another thing!



For sure dbmail-util is not cleaning messages!

I was checking the function on dm_db.c , and after a:



dbmail-util -d -y



the query:



r = db_query(c, "SELECT p.id FROM %smimeparts p LEFT JOIN
%spartlists l ON p.id = l.part_id "

"WHERE l.part_id IS NULL", DBPFX, DBPFX);



That is on function:



int db_icheck_mimeparts(gboolean cleanup)



Doesn't return any rows, I tested it manually.



Has it doesn't return any rows, the while below this where the messages with
DELETE status are supposed to be deleted by the query:



db_exec(c, "DELETE FROM %smimeparts WHERE id
= %llu", DBPFX, *(u64_t *)ids->data);



It doesn't.





Where the error may be?

I must be before this


mysql.jorge at decimal

Sep 19, 2009, 12:10 PM

Post #2 of 9 (1640 views)
Permalink
Re: dbmail-util not cleaning messages [In reply to]

Extra info on this:



mysql> UPDATE dbmail_messages SET status = 3 WHERE status = 2;

Query OK, 0 rows affected (0.00 sec)

Rows matched: 0 Changed: 0 Warnings: 0



mysql> DELETE FROM dbmail_messages WHERE status=3;

Query OK, 1 row affected (0.03 sec)



mysql> DELETE dbmail_mimeparts.* FROM dbmail_mimeparts LEFT JOIN
dbmail_partlists ON (dbmail_mimeparts.id=dbmail_partlists.part_id) WHERE
dbmail_partlists.physmessage_id IS NULL;

Query OK, 0 rows affected (13 min 31.80 sec)





The data on partlists are not being deleted, and I think this is the reason
why messages are not being deleted on function “int
db_icheck_mimeparts(gboolean cleanup)”





Could this be checked?







From: dbmail-dev-bounces [at] dbmail [mailto:dbmail-dev-bounces [at] dbmail]
On Behalf Of Jorge Bastos
Sent: sábado, 19 de Setembro de 2009 19:07
To: 'DBMAIL Developers Mailinglist'
Subject: [Dbmail-dev] dbmail-util not cleaning messages



Paul/Jon,



I ask your help on another thing!



For sure dbmail-util is not cleaning messages!

I was checking the function on dm_db.c , and after a:



dbmail-util -d –y



the query:



r = db_query(c, "SELECT p.id FROM %smimeparts p LEFT JOIN
%spartlists l ON p.id = l.part_id "

"WHERE l.part_id IS NULL", DBPFX, DBPFX);



That is on function:



int db_icheck_mimeparts(gboolean cleanup)



Doesn’t return any rows, I tested it manually.



Has it doesn’t return any rows, the while below this where the messages with
DELETE status are supposed to be deleted by the query:



db_exec(c, "DELETE FROM %smimeparts WHERE id
= %llu", DBPFX, *(u64_t *)ids->data);



It doesn’t.





Where the error may be?

I must be before this


paul at nfg

Sep 19, 2009, 1:27 PM

Post #3 of 9 (1634 views)
Permalink
Re: dbmail-util not cleaning messages [In reply to]

Jorge,

You're mixing up tables here.

there is *no* direct connection between messages and partlists. There
*is* a 1-1 connection between physmessages and partlists, and that is
maintained by a database constraint.

But mimeparts have a 1-N connection to partlists, so no database
constraint there. To remedie this we need to find lost mimeparts, and
that is what this query is for:

>
> r = db_query(c, "SELECT p.id FROM %smimeparts p LEFT
> JOIN %spartlists l ON p.id = l.part_id "
>
> "WHERE l.part_id IS NULL", DBPFX, DBPFX);
>


Unconnected mimeparts are deleted thus:

> db_exec(c, "DELETE FROM %smimeparts
> WHERE id = %llu", DBPFX, *(u64_t *)ids->data);


But what you're doing is something else:


Jorge Bastos wrote:
> mysql> UPDATE dbmail_messages SET status = 3 WHERE status = 2;
>
> Query OK, 0 rows affected (0.00 sec)
>
> Rows matched: 0 Changed: 0 Warnings: 0

So, no changes!

> mysql> DELETE FROM dbmail_messages WHERE status=3;
>
> Query OK, 1 row affected (0.03 sec)

Ok, 1 message deleted. But remember this deletes only the metadata of a
message (dbmail_messages and dbmail_keywords). The actual raw message
data are stored in [dbmail_physmessage (1) -> (1) dbmail_partlists (n)
-> (n) dbmail_mimeparts] plus all the caching tables that link to
dbmail_physmessage.

To find and delete physmessages that are no longer connected to any rows
in dbmail_messages you need to run dbmail-util -ty, which you do just fine.

Problem is dbmail-util -ty doesn't give back feedback anymore to report
the number of rows updated or deleted by -dy and -py respectively. So
there is no way for my to tell you more without more info.

However, after doing a delete on dbmail_messages dbmail-util -ty should
report cleaning up physmessages, and mimeparts, but no partlists.


> mysql> DELETE dbmail_mimeparts.* FROM dbmail_mimeparts LEFT JOIN
> dbmail_partlists ON (dbmail_mimeparts.id=dbmail_partlists.part_id) WHERE
> dbmail_partlists.physmessage_id IS NULL;

I'm pretty sure we don't use that kind of query. I wasn't even aware of
this trick.

> Query OK, 0 rows affected (13 min 31.80 sec)

Which is ok, since you haven't cleaned up the physmessage connected to
the message you deleted.

But you are onto something here, which I've fixed now.


--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


mysql.jorge at decimal

Sep 19, 2009, 2:33 PM

Post #4 of 9 (1646 views)
Permalink
Re: dbmail-util not cleaning messages [In reply to]

> > mysql> DELETE dbmail_mimeparts.* FROM dbmail_mimeparts LEFT JOIN
> > dbmail_partlists ON (dbmail_mimeparts.id=dbmail_partlists.part_id)
> WHERE
> > dbmail_partlists.physmessage_id IS NULL;
>
> I'm pretty sure we don't use that kind of query. I wasn't even aware of
> this trick.

It was something I used in 2.3.2.

Thanks for your fix!
This fixes this problem, and also something I reported about having uncached
messages all the time when running "dbmail-util -by".

Only one question,
In every run, the partlists part, always has 0 unconnected partlists's. Is
it normal? Example below.


---
Setting DELETE status for deleted messages...
Ok. Messages set for deletion.
Re-calculating used quota for all users...
Ok. Used quota updated for all users.

Maintenance done. No errors found.

Deleting messages with DELETE status...
Ok. Messages deleted.

Maintenance done. No errors found.

Repairing DBMAIL message integrity...

Repairing DBMAIL physmessage integrity...
Ok. Found [12181] unconnected physmessages.
Ok. Orphaned physmessages deleted.

Repairing DBMAIL partlists integrity...
Ok. Found [0] unconnected partlists.

Repairing DBMAIL mimeparts integrity...
Ok. Found [40492] unconnected mimeparts.
Ok. Orphaned mimeparts deleted.

Maintenance done. No errors found.

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


mysql.jorge at decimal

Sep 19, 2009, 2:48 PM

Post #5 of 9 (1627 views)
Permalink
Re: dbmail-util not cleaning messages [In reply to]

Allow me other question for this, with 40492 mimeparts to delete, or 11
mimeparts to delete also, it takes barely the same time.

Normal?




> Only one question,
> In every run, the partlists part, always has 0 unconnected partlists's.
> Is
> it normal? Example below.
>
>
> ---
> Setting DELETE status for deleted messages...
> Ok. Messages set for deletion.
> Re-calculating used quota for all users...
> Ok. Used quota updated for all users.
>
> Maintenance done. No errors found.
>
> Deleting messages with DELETE status...
> Ok. Messages deleted.
>
> Maintenance done. No errors found.
>
> Repairing DBMAIL message integrity...
>
> Repairing DBMAIL physmessage integrity...
> Ok. Found [12181] unconnected physmessages.
> Ok. Orphaned physmessages deleted.
>
> Repairing DBMAIL partlists integrity...
> Ok. Found [0] unconnected partlists.
>
> Repairing DBMAIL mimeparts integrity...
> Ok. Found [40492] unconnected mimeparts.
> Ok. Orphaned mimeparts deleted.
>
> Maintenance done. No errors found.
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev [at] dbmail
> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


paul at nfg

Sep 20, 2009, 11:41 AM

Post #6 of 9 (1614 views)
Permalink
Re: dbmail-util not cleaning messages [In reply to]

Jorge Bastos wrote:
>>> mysql> DELETE dbmail_mimeparts.* FROM dbmail_mimeparts LEFT JOIN
>>> dbmail_partlists ON (dbmail_mimeparts.id=dbmail_partlists.part_id)
>> WHERE
>>> dbmail_partlists.physmessage_id IS NULL;
>> I'm pretty sure we don't use that kind of query. I wasn't even aware of
>> this trick.
>
> It was something I used in 2.3.2.
>
> Thanks for your fix!
> This fixes this problem, and also something I reported about having uncached
> messages all the time when running "dbmail-util -by".
>
> Only one question,
> In every run, the partlists part, always has 0 unconnected partlists's. Is
> it normal? Example below.

yes. Those are flushed out by the foreign key constraint in partlists:

CONSTRAINT `dbmail_partlists_ibfk_1` FOREIGN KEY (`physmessage_id`)
REFERENCES `dbmail_physmessage` (`id`) ON DELETE CASCADE ON UPDATE CASCADE



--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


paul at nfg

Sep 20, 2009, 11:42 AM

Post #7 of 9 (1614 views)
Permalink
Re: dbmail-util not cleaning messages [In reply to]

Jorge Bastos wrote:
> Allow me other question for this, with 40492 mimeparts to delete, or 11
> mimeparts to delete also, it takes barely the same time.
>
> Normal?

Is that a question?


--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


mysql.jorge at decimal

Sep 20, 2009, 11:57 AM

Post #8 of 9 (1613 views)
Permalink
Re: dbmail-util not cleaning messages [In reply to]

> Jorge Bastos wrote:
> > Allow me other question for this, with 40492 mimeparts to delete, or
> 11
> > mimeparts to delete also, it takes barely the same time.
> >
> > Normal?
>
> Is that a question?
>

Yap, it takes almost the same time to delete a lot or a few messages.

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev


michael.monnerie at is

Sep 21, 2009, 2:48 AM

Post #9 of 9 (1597 views)
Permalink
Re: dbmail-util not cleaning messages [In reply to]

On Samstag 19 September 2009 Paul J Stevens wrote:
> Jorge,
>
> You're mixing up tables here.
>
> there is *no* direct connection between messages and partlists. There
> *is* a 1-1 connection between physmessages and partlists, and that is
> maintained by a database constraint.
>
> But mimeparts have a 1-N connection to partlists, so no database
> constraint there.

Any chance to get an ER diagram for the 2.3 series?
http://www.dbmail.org/dokuwiki/doku.php/er-model

Would be really nice to have a quick overview, and an idea of how things
work. I also used it a lot for the 2.2 series to optimize some queries.

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 65 31 .network.your.ideas.
// PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4

_______________________________________________
Dbmail-dev mailing list
Dbmail-dev [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev

DBMail dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.