Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DBMail: users

DBMail 2.3.6 released

 

 

DBMail users RSS feed   Index | Next | Previous | View Threaded


paul at nfg

Jun 28, 2009, 8:19 AM

Post #1 of 11 (2186 views)
Permalink
DBMail 2.3.6 released

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Greetings,

I've just released dbmail 2.3.6, the latest development release.

There are still some rough edges in the packaging and documentation, but
otherwise the code is approaching production level quality.

New features in this release:

Single-instance header storage

The header caching tables used since 2.2 have been replaced with a new
schema, optimized for a much smaller storage footprint, and therefor
faster access. Headers are now cached using a single-instance storage
pattern, similar to the one used for the message parts. This change also
introduces for the first time the appearance of views in the database,
which is somewhat experimental because of some uncertainties with regard
to the possible performance impact this may have.

Authentication logging

A new table was added to the schema to log a couple of key metrics for
users connecting to one of the daemons.

Of course, a number of bugs have also been fixed along the way:

* 0000689: [Command-Line programs (dbmail-users, dbmail-util)]
dbmail-exports fails with File size limit exceeded (paul) - resolved.
* 0000775: [PIPE delivery (dbmail-smtp/dbmail-deliver)] Issue with
multiple inline attachments (paul) - resolved.
* 0000783: [General] Boundary missing in message construction (paul) -
resolved.
* 0000681: [General] message reconstruction fails on message (paul) -
resolved.
* 0000774: [IMAP daemon] SQLException using dbmail-imapd - resolved.
* 0000766: [POP3 daemon] dbmail-pop3d crash (paul) - resolved.
* 0000754: [General] single instance storage for headervalues (paul) -
resolved.
* 0000760: [LMTP daemon] DNS Regresion in 2.3.5 (netvulture) - resolved.
* 0000743: [LMTP daemon] Memory leak in lmtpd (paul) - resolved.
* 0000755: [POP3 daemon] POP3D crash when fetchmail tries to connect
(paul) - resolved.
* 0000720: [Command-Line programs (dbmail-users, dbmail-util)] Missing
operations on dbmail-util (paul) - resolved.

Changelog:
http://git.dbmail.eu/cgit/cgit.cgi/paul/dbmail/commit/?id=f8e78884e36d8d1a756411c1c891e1a2b767ba17

Download:
http://git.dbmail.eu/cgit/cgit.cgi/paul/dbmail/snapshot/dbmail-2.3.6.tar.bz2


- --
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpHif8ACgkQ8iITvBH4zTHt3gCfeg4dneTRCOiEPKPSyBKHMMlv
oAIAnjb0gmXqbPZlhQ4YQpwyCeR/txwL
=AZFb
-----END PGP SIGNATURE-----
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail


mysql.jorge at decimal

Jun 28, 2009, 10:11 AM

Post #2 of 11 (2100 views)
Permalink
Re: DBMail 2.3.6 released [In reply to]

Good work Paul!

Feedback from people who will use this 2.3.6 version are welcome.

Jorge,


> -----Original Message-----
> From: dbmail-bounces [at] dbmail [mailto:dbmail-bounces [at] dbmail] On
> Behalf Of Paul J Stevens
> Sent: domingo, 28 de Junho de 2009 16:20
> To: DBMail mailinglist; DBMAIL Developers Mailinglist
> Subject: [Dbmail] DBMail 2.3.6 released
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> Greetings,
>
> I've just released dbmail 2.3.6, the latest development release.
>
> There are still some rough edges in the packaging and documentation,
> but
> otherwise the code is approaching production level quality.
>
> New features in this release:
>
> Single-instance header storage
>
> The header caching tables used since 2.2 have been replaced with a new
> schema, optimized for a much smaller storage footprint, and therefor
> faster access. Headers are now cached using a single-instance storage
> pattern, similar to the one used for the message parts. This change
> also
> introduces for the first time the appearance of views in the database,
> which is somewhat experimental because of some uncertainties with
> regard
> to the possible performance impact this may have.
>
> Authentication logging
>
> A new table was added to the schema to log a couple of key metrics for
> users connecting to one of the daemons.
>
> Of course, a number of bugs have also been fixed along the way:
>
> * 0000689: [Command-Line programs (dbmail-users, dbmail-util)]
> dbmail-exports fails with File size limit exceeded (paul) - resolved.
> * 0000775: [PIPE delivery (dbmail-smtp/dbmail-deliver)] Issue with
> multiple inline attachments (paul) - resolved.
> * 0000783: [General] Boundary missing in message construction (paul) -
> resolved.
> * 0000681: [General] message reconstruction fails on message (paul) -
> resolved.
> * 0000774: [IMAP daemon] SQLException using dbmail-imapd - resolved.
> * 0000766: [POP3 daemon] dbmail-pop3d crash (paul) - resolved.
> * 0000754: [General] single instance storage for headervalues (paul) -
> resolved.
> * 0000760: [LMTP daemon] DNS Regresion in 2.3.5 (netvulture) -
> resolved.
> * 0000743: [LMTP daemon] Memory leak in lmtpd (paul) - resolved.
> * 0000755: [POP3 daemon] POP3D crash when fetchmail tries to connect
> (paul) - resolved.
> * 0000720: [Command-Line programs (dbmail-users, dbmail-util)] Missing
> operations on dbmail-util (paul) - resolved.
>
> Changelog:
> http://git.dbmail.eu/cgit/cgit.cgi/paul/dbmail/commit/?id=f8e78884e36d8
> d1a756411c1c891e1a2b767ba17
>
> Download:
> http://git.dbmail.eu/cgit/cgit.cgi/paul/dbmail/snapshot/dbmail-
> 2.3.6.tar.bz2
>
>
> - --
> ________________________________________________________________
> Paul Stevens paul at nfg.nl
> NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
> The Netherlands________________________________http://www.nfg.nl
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkpHif8ACgkQ8iITvBH4zTHt3gCfeg4dneTRCOiEPKPSyBKHMMlv
> oAIAnjb0gmXqbPZlhQ4YQpwyCeR/txwL
> =AZFb
> -----END PGP SIGNATURE-----
> _______________________________________________
> DBmail mailing list
> DBmail [at] dbmail
> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail


vulture at netvulture

Jun 28, 2009, 11:42 AM

Post #3 of 11 (2105 views)
Permalink
Re: [Dbmail-dev] DBMail 2.3.6 released [In reply to]

Paul J Stevens wrote:
> New features in this release:
>
There is also a new option -M on dbmail-util to migrate 2.2 messageblks
stored messages to the 2.3 single-instance storage. Messages are moved
10000 per run of dbmail-util. You can change the 10000 limit with a -m #
argument.

-Jon

--
Scanned for viruses and dangerous content by MailScanner

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail


jake at vapourforge

Jun 28, 2009, 6:02 PM

Post #4 of 11 (2103 views)
Permalink
Re: DBMail 2.3.6 released [In reply to]

Any chance of debs of this?
It makes upgrading much easier ;->

Oh and is there anything to migrate headers over to the new storage as well?
Paul J Stevens wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> Greetings,
>
> I've just released dbmail 2.3.6, the latest development release.
>
> There are still some rough edges in the packaging and documentation, but
> otherwise the code is approaching production level quality.
>
> New features in this release:
>
> Single-instance header storage
>
> The header caching tables used since 2.2 have been replaced with a new
> schema, optimized for a much smaller storage footprint, and therefor
> faster access. Headers are now cached using a single-instance storage
> pattern, similar to the one used for the message parts. This change also
> introduces for the first time the appearance of views in the database,
> which is somewhat experimental because of some uncertainties with regard
> to the possible performance impact this may have.
>
> Authentication logging
>
> A new table was added to the schema to log a couple of key metrics for
> users connecting to one of the daemons.
>
> Of course, a number of bugs have also been fixed along the way:
>
> * 0000689: [Command-Line programs (dbmail-users, dbmail-util)]
> dbmail-exports fails with File size limit exceeded (paul) - resolved.
> * 0000775: [PIPE delivery (dbmail-smtp/dbmail-deliver)] Issue with
> multiple inline attachments (paul) - resolved.
> * 0000783: [General] Boundary missing in message construction (paul) -
> resolved.
> * 0000681: [General] message reconstruction fails on message (paul) -
> resolved.
> * 0000774: [IMAP daemon] SQLException using dbmail-imapd - resolved.
> * 0000766: [POP3 daemon] dbmail-pop3d crash (paul) - resolved.
> * 0000754: [General] single instance storage for headervalues (paul) -
> resolved.
> * 0000760: [LMTP daemon] DNS Regresion in 2.3.5 (netvulture) - resolved.
> * 0000743: [LMTP daemon] Memory leak in lmtpd (paul) - resolved.
> * 0000755: [POP3 daemon] POP3D crash when fetchmail tries to connect
> (paul) - resolved.
> * 0000720: [Command-Line programs (dbmail-users, dbmail-util)] Missing
> operations on dbmail-util (paul) - resolved.
>
> Changelog:
> http://git.dbmail.eu/cgit/cgit.cgi/paul/dbmail/commit/?id=f8e78884e36d8d1a756411c1c891e1a2b767ba17
>
> Download:
> http://git.dbmail.eu/cgit/cgit.cgi/paul/dbmail/snapshot/dbmail-2.3.6.tar.bz2
>
>
> - --
> ________________________________________________________________
> Paul Stevens paul at nfg.nl
> NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
> The Netherlands________________________________http://www.nfg.nl
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkpHif8ACgkQ8iITvBH4zTHt3gCfeg4dneTRCOiEPKPSyBKHMMlv
> oAIAnjb0gmXqbPZlhQ4YQpwyCeR/txwL
> =AZFb
> -----END PGP SIGNATURE-----
> _______________________________________________
> DBmail mailing list
> DBmail [at] dbmail
> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
>


--

Vapour Forge

Jake Anderson

Project Manager

Mobile: 0412 897 125

Email: jake [at] vapourforge

Web Page: www.vapourforge.com

Your source for custom IT services
Attachments: vf.gif (1.45 KB)


vulture at netvulture

Jun 28, 2009, 9:15 PM

Post #5 of 11 (2101 views)
Permalink
Re: DBMail 2.3.6 released [In reply to]

Jake Anderson wrote:
> Any chance of debs of this?
> It makes upgrading much easier ;->
>
> Oh and is there anything to migrate headers over to the new storage as
> well?
Debs would be Paul's dept.

Part of upgrading to 2.3.6 removes the old header caching tables and
creates new ones. dbmail-util then needs to be run to finishing the
upgrade by rebuilding the cache which will be in the new single-instance
header caching. You can further finish up with migrating 2.2.x stored
messages with the -M option to move those messages into the new
single-instance storage of mimeparts.

-Jon

--
Scanned for viruses and dangerous content by MailScanner

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail


pwadas at jewish

Jul 2, 2009, 4:38 AM

Post #6 of 11 (2093 views)
Permalink
Re: DBMail 2.3.6 released - for which version is psql create tables script issued with tgz? [In reply to]

I guess I shoud make some x.y.z_to_a.b.c.sql script, but from which version
I should start?
I deleted all tables in pgsql, and now I'm trying to recreate it with 2.3.6
package

regards,
Piotr
--
View this message in context: http://www.nabble.com/DBMail-2.3.6-released-tp24242394p24305257.html
Sent from the dbmail users mailing list archive at Nabble.com.

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail


pwadas at jewish

Jul 2, 2009, 4:38 AM

Post #7 of 11 (2100 views)
Permalink
Re: DBMail 2.3.6 released - for which version is psql create tables script issued with tgz? [In reply to]

I guess I shoud make some x.y.z_to_a.b.c.sql script, but from which version I
should start?
I deleted all tables in pgsql, and now I'm trying to recreate it with sql
script from 2.3.6 package

regards,
Piotr
--
View this message in context: http://www.nabble.com/DBMail-2.3.6-released-tp24242394p24305257.html
Sent from the dbmail users mailing list archive at Nabble.com.

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail


paul at nfg

Jul 2, 2009, 12:48 PM

Post #8 of 11 (2087 views)
Permalink
Re: DBMail 2.3.6 released - for which version is psql create tables script issued with tgz? [In reply to]

Piotr Wadas wrote:
>
> I guess I shoud make some x.y.z_to_a.b.c.sql script, but from which version
> I should start?

which version do you have installed?

> I deleted all tables in pgsql, and now I'm trying to recreate it with 2.3.6
> package

try:

sql/postgresql/create_tables.pgsql



--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail


pwadas at jewish

Jul 8, 2009, 1:02 AM

Post #9 of 11 (2058 views)
Permalink
Re: DBMail 2.3.6 released ( works fine ) [In reply to]

Hello,
I installed 2.3.6 from packages provided by Paul, and recreated tables from
the sql
script you've pointed me to. Seems to work fine everything, except

1) To purge after message delete, I issue a command

dbmail-util -ctubpdyv

it works actually, but after third call - I need to issue this
command about three times, to get rid of messages finally,
first and second call has no effect. Is it related to the construction

- set delete status
- then purge (?)

shall I issue with "set delete status" switch only, and then with "purge
deleted" switch
only? order of switches I used (-ctubpdyv ) has a meaning? Only with third
call I got expected info "found/cleaned up 1234 mimeparts"

2) after setting delete status and purging messages, there's still some
low-size
garbage in dbmail_headervalue table, undoubtedly related to purged messages
(according to header's content), so the headers are orphaned, which column
should I match in sql join to clean up dbmail_headervalue table ?

3) after setting delete status and purging messages with dbmail-util, I
still need
to issue vacuumdb -U pgadminuser dbmail-database, to regain real disk space,
although dbmail-util information (-v) messages say it does some vacuuming on
its own.

Regards,
Piotr Wadas
--
View this message in context: http://www.nabble.com/DBMail-2.3.6-released-tp24242394p24387146.html
Sent from the dbmail users mailing list archive at Nabble.com.

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail


pwadas at jewish

Jul 8, 2009, 1:43 PM

Post #10 of 11 (2053 views)
Permalink
Re: DBMail 2.3.6 released ( works fine ) [In reply to]

actually this is what left after delete/purge/vacuum (PG 8.3)
dbmail_users, dbmail_subscriptions, and dbmail_mailboxes is reasonable,
but I expect headernames and header values to be empty too.

table_name | how_many_rows
------------------------+------
dbmail_aliases | 0
dbmail_authlog | 0
dbmail_envelope | 0
dbmail_filters | 0
dbmail_header | 0
dbmail_mimeparts | 0
dbmail_pbsp | 0
dbmail_physmessage | 0
dbmail_referencesfield | 0
dbmail_replycache | 0
dbmail_sievescripts | 0
dbmail_usermap | 0
dbmail_keywords | 0
dbmail_messageblks | 0
dbmail_messages | 0
dbmail_acl | 0
dbmail_partlists | 0
dbmail_users | 5
dbmail_mailboxes | 6
dbmail_subscription | 6
dbmail_headername | 58
dbmail_headervalue | 2052
(24 rows)

--
View this message in context: http://www.nabble.com/DBMail-2.3.6-released-tp24242394p24398973.html
Sent from the dbmail users mailing list archive at Nabble.com.

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail


vulture at netvulture

Jul 9, 2009, 9:56 AM

Post #11 of 11 (2044 views)
Permalink
Re: DBMail 2.3.6 released ( works fine ) [In reply to]

Something is missing in the constraints or dbmail-util clean up. With 0
messages, then headernames/headervalues should be 0 as well. I'm glad to
see that headers is 0 as it should be.

These should clean up the extra rows. Please note that these will take a
while on a database with a lot of header caching and messages still
present. Took 15 mins to kill 36899 rows (5037478 rows remain) for just
the first one for me (mysql). Second one took only 25 seconds to kill
241 rows (5328 rows remain). I have 492531 physmessages.

DELETE FROM dbmail_headervalue where id NOT IN (SELECT
DISTINCT(headervalue_id) FROM dbmail_header);
DELETE FROM dbmail_headername where id NOT IN (SELECT
DISTINCT(headername_id) FROM dbmail_header);


-Jon

Piotr Wadas wrote:
> actually this is what left after delete/purge/vacuum (PG 8.3)
> dbmail_users, dbmail_subscriptions, and dbmail_mailboxes is reasonable,
> but I expect headernames and header values to be empty too.
>
> table_name | how_many_rows
> ------------------------+------
> dbmail_aliases | 0
> dbmail_authlog | 0
> dbmail_envelope | 0
> dbmail_filters | 0
> dbmail_header | 0
> dbmail_mimeparts | 0
> dbmail_pbsp | 0
> dbmail_physmessage | 0
> dbmail_referencesfield | 0
> dbmail_replycache | 0
> dbmail_sievescripts | 0
> dbmail_usermap | 0
> dbmail_keywords | 0
> dbmail_messageblks | 0
> dbmail_messages | 0
> dbmail_acl | 0
> dbmail_partlists | 0
> dbmail_users | 5
> dbmail_mailboxes | 6
> dbmail_subscription | 6
> dbmail_headername | 58
> dbmail_headervalue | 2052
> (24 rows)
>
>


--
Scanned for viruses and dangerous content by MailScanner

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

DBMail users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.