jake at vapourforge
Nov 17, 2008, 7:27 AM
Post #6 of 9
(1343 views)
Permalink
|
>
>
> My plan is to do that nifty trick with the copying over of the user
> tables then point the port forward at the new host so we don't drop
> any email.
> Then I'm looking at the "best" way of moving the bulk of the email
> over. I don't want to just mysqldump it over, I want it all in the
> single instance storage system.
>
> Any recommendations? The last few times I have tried imapcopy and
> imapsync they seem to crash or cause dbmail to crash or something
> equally silly (not to mention they are slow as a wet week). The dump
> to mbox tool lost/merged or some such a few emails last time I tried
> it as i recall although it was nice and fast.
Well it looks like the dbmail mbox exporter and the mbox-import tool
like to play nicley now, I copied over a gig worth of email to test the
new server and it seems pretty good. I did have imap crash though with
no error in dbmail.err or anything in dbmail.log however this was in
mail.log
Nov 17 03:00:09 mail dbmail/maintenance[12270]: [0x9f60958]
Warning:[db] db_set_headercache(+1604): error caching headers for
physmessage: [11406]
Nov 17 03:00:09 mail dbmail/maintenance[12270]: [0x9f60958] Warning:[db]
db_set_headercache(+1604): error caching headers for physmessage: [9262]
Nov 17 03:00:09 mail dbmail/maintenance[12270]: [0x9f60958] Warning:[db]
db_set_headercache(+1604): error caching headers for physmessage: [9246]
Nov 17 03:16:40 mail dbmail/deliver[7708]: [0x95ee9c0] Error:[message]
_header_cache(+1510): SQLException: Duplicate entry
'103090-9-xxxxxxxx.xxxxxxx [at] cba' for key 3
Nov 17 03:18:13 mail dbmail/deliver[12193]: [0x9f429c0] Error:[message]
_header_cache(+1510): SQLException: Duplicate entry
'105016-9-xxxxxxxx.xxxxxxx [at] cba' for key 3
That might be my fault for loading the data several times but it seems
unlikely as its still loaded through dbmail-deliver.
Whilst on the subject of loading data I made a frontend for mbox2dbmail
that should load all the accounts and mailboxes dumped by a wildcard
dump from the source system and as a bonus do it in parallel (ie 3
simultaneous insertion programs).
Its tailored pretty specifically to the setup i was using and isn't
general purpose in anyway yet however it was useful to me ;->
basically copy the results of your dbmail-export -u *
to a directory called "migrate" on the target system
(so ~/migrate/jake [at] spamthis would be a result)
put this program thus named migrate2dbmail.py version 0.01 and
mbox2dbmail in the parent directory (IE ~)
set how many threads you would like to use to insert (i used 3 because
thats how many CPU's i gave the VM)
then run it
python migrate2dbmail.py
It should spit out N text files for you
thread-1 thread-2 thread-3
with the commands to run mbox2dbmail in it.
so chmod +x thread*
screen (its probably going to take a while)
(./thread-1 &) ; (./thread-2 &) ; (./thread-3 &)
Its not tuned in any way shape or form it just splits the mbox files
directly into the 3 processes but its better than nothing.
It got me a pretty linear improvement in performance, I was hitting
~240% CPU use by dbmail-deliver 10% by python and 50% or so by mysql.
The disk was pretty relaxed, nowhere near hitting that. I estimate it
was inserting 5-10 emails a second or so, into a paravirtualised machine
on a quad core host with a generic 7200RPM 80GB drive as the single disk.
The emails themselves start appearing a minute or 2 after the thing says
they were inserted so don't get too impatient ;->
==========================================================
# Copyright (C) 2008 Jake Anderson <jake at vapourforge dot com>
# migrate2dbmail Version 0.01
#
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either
# version 2 of the License, or (at your option) any later
# version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
#
import os
import re
from os.path import join, getsize
os.chdir('migrate/')
numthreads = 3
outputfile = {}
for i in range(1,numthreads+1):
outputfile[i] = ''
currthread = 0
for root, dirs, files in os.walk('.'):
currthread += 1
if currthread > numthreads:
currthread = 1
print currthread
if root.count('/') == 1:
#this is the top level folder, set the account name
curract = root.split('/')[1]
for file in files:
mboxpath = os.path.abspath(os.path.join(root,file))
dest_path = '/'.join(root.split('/')[2:])+'/'+file[:-5]
if dest_path[0:1] == "/":
dest_path = dest_path[1:]
# print "python mbox2dbmail -p /usr/sbin/dbmail-deliver -t mbox
-u \"%s\" -m \"%s\" -b \"%s\"" % (curract,mboxpath,dest_path)
outputfile[currthread] += "python mbox2dbmail -p
/usr/sbin/dbmail-deliver -t mbox -u \"%s\" -m \"%s\" -b \"%s\"\n" %
(curract,mboxpath,dest_path)
# print dest_path
os.chdir('..')
for i in range(1,numthreads+1):
f = open("thread-%s" % (i),'w')
f.write(outputfile[i])
_______________________________________________
DBmail mailing list
DBmail [at] dbmail
https://mailman.fastxs.nl/mailman/listinfo/dbmail
|
