Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DBMail: users

Which tables

 

 

DBMail users RSS feed   Index | Next | Previous | View Threaded


curtis at maurand

Oct 27, 2008, 5:58 AM

Post #1 of 11 (1752 views)
Permalink
Which tables

Hello all,

I find this list to be incredibly helpful. Thank you all in advance of
my question.

I need to move my installation from the current server to another. That
could be accomplished by simply doing a mysqldump -u <user> -p<pass>
--database dbmail |mysql -u<user> -p<pass> -h <new server> dmail

That's known and been done and that's great, but I don't want to move
several GB of mail. I only want to transfer the user accounts, mailbox
structure and sieve scripts. Which tables do I need to copy to the new
server? I'm running 2.2.9 on the older machine and 2.2.10 on the newer
unit.

Thanks, again,
Curtis Maurand
xyonet.com


dbmail at tech

Oct 27, 2008, 7:23 AM

Post #2 of 11 (1674 views)
Permalink
Re: Which tables [In reply to]

Curtis Maurand wrote:
> Hello all,
>
> I find this list to be incredibly helpful. Thank you all in advance
> of my question.
>
> I need to move my installation from the current server to another.
> That could be accomplished by simply doing a mysqldump -u <user>
> -p<pass> --database dbmail |mysql -u<user> -p<pass> -h <new server> dmail
>
> That's known and been done and that's great, but I don't want to move
> several GB of mail. I only want to transfer the user accounts,
> mailbox structure and sieve scripts. Which tables do I need to copy
> to the new server? I'm running 2.2.9 on the older machine and 2.2.10
> on the newer unit.
Hi,

The main large table is the dbmail_messageblks, which stores the bulk of
the email- but this does rely on an up-to-date dbmail_messages,
dbmail_phymessage and lots of the other tables which are used for
caching I believe? I assume you wish to test your installation first? If
this is the case, you should be able to copy over everything except the
messageblk. Recreate the messageblk on the new server and run
dbmail-util to tidy up your installation.

By the way - you may find it faster to use the
mk-parallel-dump/mk-parallel-restore in the mysql tool kit -
http://www.maatkit.org/ than a single instance of mysqldump

HTH

SG


paul at nfg

Oct 27, 2008, 7:57 AM

Post #3 of 11 (1683 views)
Permalink
Re: Which tables [In reply to]

Curtis Maurand wrote:
> Hello all,
>
> I find this list to be incredibly helpful. Thank you all in advance of
> my question.
>
> I need to move my installation from the current server to another. That
> could be accomplished by simply doing a mysqldump -u <user> -p<pass>
> --database dbmail |mysql -u<user> -p<pass> -h <new server> dmail
>
> That's known and been done and that's great, but I don't want to move
> several GB of mail. I only want to transfer the user accounts, mailbox
> structure and sieve scripts. Which tables do I need to copy to the new
> server? I'm running 2.2.9 on the older machine and 2.2.10 on the newer
> unit.

Why? It's not obvious what you're trying to achieve here.


--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
https://mailman.fastxs.nl/mailman/listinfo/dbmail


curtis at maurand

Oct 27, 2008, 11:24 AM

Post #4 of 11 (1671 views)
Permalink
Re: Which tables [In reply to]

I'm trying to move user accounts and mailbox structure without moving
the messages. I have bandwidth limits not the least of which is that
the current system is residing on a T1, is several GB in size (which
means hours to transfer) and I have bandwidth and diskspace limits on
the new host.

Thanks,
Curtis

Paul J Stevens wrote:
> Curtis Maurand wrote:
>
>> Hello all,
>>
>> I find this list to be incredibly helpful. Thank you all in advance of
>> my question.
>>
>> I need to move my installation from the current server to another. That
>> could be accomplished by simply doing a mysqldump -u <user> -p<pass>
>> --database dbmail |mysql -u<user> -p<pass> -h <new server> dmail
>>
>> That's known and been done and that's great, but I don't want to move
>> several GB of mail. I only want to transfer the user accounts, mailbox
>> structure and sieve scripts. Which tables do I need to copy to the new
>> server? I'm running 2.2.9 on the older machine and 2.2.10 on the newer
>> unit.
>>
>
> Why? It's not obvious what you're trying to achieve here.
>
>
>


paul at nfg

Oct 27, 2008, 1:19 PM

Post #5 of 11 (1670 views)
Permalink
Re: Which tables [In reply to]

Curtis Maurand wrote:
>
> I'm trying to move user accounts and mailbox structure without moving
> the messages.

understood. Users will suddenly start seeing empty folders.

very tricky.

If you carefully setup all tables and dump/reload all tables *except*
dbmail_messageblks you will avoid transferring the bulk of your data
while retaining user experience.

After setting up the new host users will start getting errors if the try
to retrieve an /old/ message until you transfer the messageblks table as
well.

You must take special care to setup the auto-increment values correctly
for the empty messageblks table. So use 'mysqldump -h oldhost -d
dbmail|mysql -h newhost dbmail' if you ever mean to transfer the old
email later on.

I've never actually done this myself, but these migration scenarios are
perfectly valid and deserve some exploratory experimentation, so please
keep us informed :-)

Of course if you don't mind if users suddenly start seeing empty folders
you can also skip:

| dbmail_ccfield |
| dbmail_datefield |
| dbmail_envelope |
| dbmail_fromfield |
| dbmail_headername |
| dbmail_headervalue |
| dbmail_mailboxes |
| dbmail_messages |
| dbmail_physmessage |
| dbmail_referencesfield |
| dbmail_replytofield |
| dbmail_subjectfield |
| dbmail_tofield |






> I have bandwidth limits not the least of which is that
> the current system is residing on a T1, is several GB in size (which
> means hours to transfer) and I have bandwidth and diskspace limits on
> the new host.
>
> Thanks,
> Curtis
>
> Paul J Stevens wrote:
>> Curtis Maurand wrote:
>>
>>> Hello all,
>>>
>>> I find this list to be incredibly helpful. Thank you all in advance of
>>> my question.
>>>
>>> I need to move my installation from the current server to another. That
>>> could be accomplished by simply doing a mysqldump -u <user> -p<pass>
>>> --database dbmail |mysql -u<user> -p<pass> -h <new server> dmail
>>>
>>> That's known and been done and that's great, but I don't want to move
>>> several GB of mail. I only want to transfer the user accounts, mailbox
>>> structure and sieve scripts. Which tables do I need to copy to the new
>>> server? I'm running 2.2.9 on the older machine and 2.2.10 on the newer
>>> unit.
>>>
>>
>> Why? It's not obvious what you're trying to achieve here.
>>
>>
>>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> DBmail mailing list
> DBmail [at] dbmail
> https://mailman.fastxs.nl/mailman/listinfo/dbmail


--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
https://mailman.fastxs.nl/mailman/listinfo/dbmail


vulture at netvulture

Oct 27, 2008, 6:16 PM

Post #6 of 11 (1667 views)
Permalink
Re: Which tables [In reply to]

One option to migrate to a new server and still retain the old messages,
but with a delay, is to start your new database with an auto_increment
number for the dbmail_messages table that is greater than the
max(message_idnr) on your existing dbmail_messages table. Then you can
import a dump (use dbmail-util -d -p -y twice to avoid coping deleted
messages) with out the table drop/create for the tables Paul mentioned
that could be skipped. Once you have imported all of the old data, even
after new messages have been delivered, simply do an update of
dbmail_messages to change the message_idnr to start an increment from a
number higher than the current max(message_idnr) on the new database.
Finally run a dbmail-util -ay twice to make sure all the cache tables
are golden. IMAP clients should see all new headers for downloading
(even messages that were seen during the data move process). Pop3
clients using leave a copy on the server may or may not have issues with
exiting messages that were already downloaded. If nothing else, all the
messages would be seen as new and needed to be downloaded.

The update process for dbmail_messages
mysql dbmail #connect to db
+-------------------+
| MAX(message_idnr) |
+-------------------+
| 31252 |
+-------------------+
1 row in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 11252 rows affected (3.33 sec)
Rows matched: 11252 Changed: 11252 Warnings: 0

+-------------------+
| MAX(message_idnr) |
+-------------------+
| 111252 |
+-------------------+
1 row in set (0.00 sec)

-Jon

Paul J Stevens wrote:
> Curtis Maurand wrote:
>
>> I'm trying to move user accounts and mailbox structure without moving
>> the messages.
>>
>
> understood. Users will suddenly start seeing empty folders.
>
> very tricky.
>
> If you carefully setup all tables and dump/reload all tables *except*
> dbmail_messageblks you will avoid transferring the bulk of your data
> while retaining user experience.
>
> After setting up the new host users will start getting errors if the try
> to retrieve an /old/ message until you transfer the messageblks table as
> well.
>
> You must take special care to setup the auto-increment values correctly
> for the empty messageblks table. So use 'mysqldump -h oldhost -d
> dbmail|mysql -h newhost dbmail' if you ever mean to transfer the old
> email later on.
>
> I've never actually done this myself, but these migration scenarios are
> perfectly valid and deserve some exploratory experimentation, so please
> keep us informed :-)
>
> Of course if you don't mind if users suddenly start seeing empty folders
> you can also skip:
>
> | dbmail_ccfield |
> | dbmail_datefield |
> | dbmail_envelope |
> | dbmail_fromfield |
> | dbmail_headername |
> | dbmail_headervalue |
> | dbmail_mailboxes |
> | dbmail_messages |
> | dbmail_physmessage |
> | dbmail_referencesfield |
> | dbmail_replytofield |
> | dbmail_subjectfield |
> | dbmail_tofield |
>
>
>
>
>
>
>
>> I have bandwidth limits not the least of which is that
>> the current system is residing on a T1, is several GB in size (which
>> means hours to transfer) and I have bandwidth and diskspace limits on
>> the new host.
>>
>> Thanks,
>> Curtis
>>
>> Paul J Stevens wrote:
>>
>>> Curtis Maurand wrote:
>>>
>>>
>>>> Hello all,
>>>>
>>>> I find this list to be incredibly helpful. Thank you all in advance of
>>>> my question.
>>>>
>>>> I need to move my installation from the current server to another. That
>>>> could be accomplished by simply doing a mysqldump -u <user> -p<pass>
>>>> --database dbmail |mysql -u<user> -p<pass> -h <new server> dmail
>>>>
>>>> That's known and been done and that's great, but I don't want to move
>>>> several GB of mail. I only want to transfer the user accounts, mailbox
>>>> structure and sieve scripts. Which tables do I need to copy to the new
>>>> server? I'm running 2.2.9 on the older machine and 2.2.10 on the newer
>>>> unit.
>>>>
>>>>
>>> Why? It's not obvious what you're trying to achieve here.
>>>
>>>
>>>
>>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> DBmail mailing list
>> DBmail [at] dbmail
>> https://mailman.fastxs.nl/mailman/listinfo/dbmail
>>
>
>
>


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
https://mailman.fastxs.nl/mailman/listinfo/dbmail


curtis at maurand

Oct 28, 2008, 2:31 PM

Post #7 of 11 (1649 views)
Permalink
Re: Which tables [In reply to]

Thank you to all. I hope to do the move Friday night. I need to do a
couple of test runs before hand. I'll keep you posted.

thanks,
Curtis

Jonathan Feally wrote:
> One option to migrate to a new server and still retain the old
> messages, but with a delay, is to start your new database with an
> auto_increment number for the dbmail_messages table that is greater
> than the max(message_idnr) on your existing dbmail_messages table.
> Then you can import a dump (use dbmail-util -d -p -y twice to avoid
> coping deleted messages) with out the table drop/create for the tables
> Paul mentioned that could be skipped. Once you have imported all of
> the old data, even after new messages have been delivered, simply do
> an update of dbmail_messages to change the message_idnr to start an
> increment from a number higher than the current max(message_idnr) on
> the new database. Finally run a dbmail-util -ay twice to make sure all
> the cache tables are golden. IMAP clients should see all new headers
> for downloading (even messages that were seen during the data move
> process). Pop3 clients using leave a copy on the server may or may not
> have issues with exiting messages that were already downloaded. If
> nothing else, all the messages would be seen as new and needed to be
> downloaded.
>
> The update process for dbmail_messages
> mysql dbmail #connect to db
> mysql> SELECT MAX(message_idnr) FROM dbmail_messages;
> +-------------------+
> | MAX(message_idnr) |
> +-------------------+
> | 31252 |
> +-------------------+
> 1 row in set (0.00 sec)
>
> mysql> set @x:=100000;
> Query OK, 0 rows affected (0.00 sec)
>
> mysql> UPDATE dbmail_messages SET message_idnr=@x:=@x+1;
> Query OK, 11252 rows affected (3.33 sec)
> Rows matched: 11252 Changed: 11252 Warnings: 0
>
> mysql> SELECT MAX(message_idnr) FROM dbmail_messages;
> +-------------------+
> | MAX(message_idnr) |
> +-------------------+
> | 111252 |
> +-------------------+
> 1 row in set (0.00 sec)
>
> -Jon
>
> Paul J Stevens wrote:
>> Curtis Maurand wrote:
>>
>>> I'm trying to move user accounts and mailbox structure without moving
>>> the messages.
>>>
>>
>> understood. Users will suddenly start seeing empty folders.
>>
>> very tricky.
>>
>> If you carefully setup all tables and dump/reload all tables *except*
>> dbmail_messageblks you will avoid transferring the bulk of your data
>> while retaining user experience.
>>
>> After setting up the new host users will start getting errors if the try
>> to retrieve an /old/ message until you transfer the messageblks table as
>> well.
>>
>> You must take special care to setup the auto-increment values correctly
>> for the empty messageblks table. So use 'mysqldump -h oldhost -d
>> dbmail|mysql -h newhost dbmail' if you ever mean to transfer the old
>> email later on.
>>
>> I've never actually done this myself, but these migration scenarios are
>> perfectly valid and deserve some exploratory experimentation, so please
>> keep us informed :-)
>>
>> Of course if you don't mind if users suddenly start seeing empty folders
>> you can also skip:
>>
>> | dbmail_ccfield |
>> | dbmail_datefield |
>> | dbmail_envelope |
>> | dbmail_fromfield |
>> | dbmail_headername |
>> | dbmail_headervalue |
>> | dbmail_mailboxes |
>> | dbmail_messages |
>> | dbmail_physmessage |
>> | dbmail_referencesfield |
>> | dbmail_replytofield |
>> | dbmail_subjectfield |
>> | dbmail_tofield |
>>
>>
>>
>>
>>
>>
>>
>>> I have bandwidth limits not the least of which is that
>>> the current system is residing on a T1, is several GB in size (which
>>> means hours to transfer) and I have bandwidth and diskspace limits on
>>> the new host.
>>>
>>> Thanks,
>>> Curtis
>>>
>>> Paul J Stevens wrote:
>>>
>>>> Curtis Maurand wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> I find this list to be incredibly helpful. Thank you all in
>>>>> advance of
>>>>> my question.
>>>>>
>>>>> I need to move my installation from the current server to
>>>>> another. That
>>>>> could be accomplished by simply doing a mysqldump -u <user> -p<pass>
>>>>> --database dbmail |mysql -u<user> -p<pass> -h <new server> dmail
>>>>>
>>>>> That's known and been done and that's great, but I don't want to move
>>>>> several GB of mail. I only want to transfer the user accounts,
>>>>> mailbox
>>>>> structure and sieve scripts. Which tables do I need to copy to
>>>>> the new
>>>>> server? I'm running 2.2.9 on the older machine and 2.2.10 on the
>>>>> newer
>>>>> unit.
>>>>>
>>>> Why? It's not obvious what you're trying to achieve here.
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> DBmail mailing list
>>> DBmail [at] dbmail
>>> https://mailman.fastxs.nl/mailman/listinfo/dbmail
>>>
>>
>>
>>
>
>
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
https://mailman.fastxs.nl/mailman/listinfo/dbmail


curtis at maurand

Oct 31, 2008, 11:14 AM

Post #8 of 11 (1641 views)
Permalink
Re: which tables [In reply to]

Hello,

the migration is complete. I moved the following tables and everything
worked out just fine. If I need to get the messages back in, I'll load
up dbmail on a test server and use IMAPsync to transfer the messages.
I've got a dump of the original database.

To move structure only, move the following:

dbmail_users
dbmail_acl
dbmail_mailboxes
dbmail_aliases
dbmail_pbsp
dbmail_sievescripts
dbmail_subscription

and if you're using dbmailadministrator (slower than death)

DBMA_MTA
DBMA_MTA_ACCESS

Hope this helps anyone who might need to move structure and not messages.

I then ran mysqlcheck -r --all_databases to fix up indexes and such.
Its working well.

Thanks for all of your help.

Curtis


paul at nfg

Oct 31, 2008, 1:24 PM

Post #9 of 11 (1638 views)
Permalink
Re: Re: which tables [In reply to]

Curtis Maurand wrote:
> Hello,
>
> the migration is complete. I moved the following tables and everything
> worked out just fine. If I need to get the messages back in, I'll load
> up dbmail on a test server and use IMAPsync to transfer the messages.

don't use imapsync for that! If you setup the other tables with the
correct auto_increment values like I suggested, you can probably simply
transfer and load dumps of the other tables. Much more efficient than
imapsync.

> I've got a dump of the original database.


--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
https://mailman.fastxs.nl/mailman/listinfo/dbmail


vulture at netvulture

Oct 31, 2008, 2:50 PM

Post #10 of 11 (1630 views)
Permalink
Re: Re: which tables [In reply to]

I thought I suggested that??? If you want all the messages in the old
db, then the auto_increment will be much faster and more reliable. If
you are wanting to filter out some messages such as only transfer mail
that is <1 year old, then imapsync would be a better choice for that. If
you already have mail in your new db, you can use the mysql update to
move those new messages to a new number. You will also want to update
you phsmessage_id to a number bigger than the largest on the old db.

-Jon

Paul Stevens wrote:
> Curtis Maurand wrote:
>> Hello,
>>
>> the migration is complete. I moved the following tables and
>> everything worked out just fine. If I need to get the messages back
>> in, I'll load up dbmail on a test server and use IMAPsync to transfer
>> the messages.
>
> don't use imapsync for that! If you setup the other tables with the
> correct auto_increment values like I suggested, you can probably
> simply transfer and load dumps of the other tables. Much more
> efficient than imapsync.
>
>> I've got a dump of the original database.
>
>


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
DBmail mailing list
DBmail [at] dbmail
https://mailman.fastxs.nl/mailman/listinfo/dbmail


curtis at maurand

Oct 31, 2008, 6:04 PM

Post #11 of 11 (1638 views)
Permalink
RE: Re: which tables [In reply to]

I'm going to make the mail available on aslow connection. If my users want the mail, they can go to that machine to get it. I'm pretty sure that I was the only person using imap or webmail (which uses imap). There are only 30 or so mailboxes on the server.

-----Original Message-----
From: "Jonathan Feally" <vulture [at] netvulture>
To: "DBMail mailinglist" <dbmail [at] dbmail>
Sent: 10/31/2008 5:50 PM
Subject: Re: [Dbmail] Re: which tables

I thought I suggested that??? If you want all the messages in the old
db, then the auto_increment will be much faster and more reliable. If
you are wanting to filter out some messages such as only transfer mail
that is <1 year old, then imapsync would be a better choice for that. If
you already have mail in your new db, you can use the mysql update to
move those new messages to a new number. You will also want to update
you phsmessage_id to a number bigger than the largest on the old
[truncated by sender]
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
https://mailman.fastxs.nl/mailman/listinfo/dbmail

DBMail users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.