dbmail-list at r
Aug 13, 2011, 7:50 PM
Post #5 of 19
On 8/13/2011 10:57 AM, Paul J Stevens wrote:
Re: Very slow imap4 performance seen when importing mail. Possible config problem?
[In reply to]
> I'll take your word for it.
== Dovecot 2.0.13 ==
MDBOX (no SiS):
== mdbox w/attachment SiS (sis posix mode, SHA1) ==
MailDir (ext4 + dir_index, no SiS attachment scheme):
> How many concurrent IMAP connections are you using?
1, single-threaded. I start testing very "simple", and work up to multi-threaded/multi-user slams once I establish a functional baseline.
I "preload" all of the imported mailboxes into RAM via cat * > /dev/null to reduce the "read cost" to near zero.
> A load below 1 indeed sounds suspicious.
I should have obtained mysql process status, but I only saw one mysqld worker process during the operation.
Should I configure innodb differently?
Reindl Harald wrote:
> if it is a option for you to upgrade mysql to 5.5
> i would strongly recommend this, our dbmail.machines
> became a hughe perofrmance boost against 5.1.x
I'll try this on the bare metal system if I don't see much improvement, thanks Reindl. I'll give your my.cnf settings a try as well, much appreciated.
> That number is about retrieval. And then only on POP3. And a very old
> number (dbmail-1.0?).
> I'd be curious about more up-to-date numbers.
I'd be more than happy to provide some once I've gotten things working properly.
Do you have a performance testing framework or set of scripts? I have the ImapTest suite, but that's more for validating compliance with the imap4r1 spec than performance. I know about MSTONE http://mstone.sourceforge.net/ but a tool that targets areas that developers suspect/worry are weak would seen sensible.
> Also, I'm not at all familiar with IO performance on VMWare, but on Xen
> IO really lags behind bare-iron databases. And thats no joke.
I have the pvscsi, vmxnet3, and other "driver helpers", and support for a paravirtualised kernel/timers, etc running and actually see very good I/O performance.
I know hdparm -t -T isn't a real disk I/O performance tool, but:
/dev/sda: <-- /tmp lives here
Timing cached reads: 7902 MB in 1.97 seconds = 4005.51 MB/sec
Timing buffered disk reads: 182 MB in 3.03 seconds = 60.02 MB/sec
/dev/sdb: <-- this is the MySQL spindle
Timing cached reads: 7702 MB in 1.97 seconds = 3901.96 MB/sec
Timing buffered disk reads: 236 MB in 3.00 seconds = 78.63 MB/sec
These are 3-4 year old Samsung 500GB drives, and this performance is about right when compared to what the host experiences natively. They're all formatted with ext4 (implying has_journal, extent, huge_file, flex_bg, uninit_bg, dir_nlink, extra_isize, sparse_super, filetype, resize_inode, dir_index, ext_attr)
> I need to keep thread-concurrency at 1 (read:
> one) to keep throughput at acceptable levels with IMAP concurrencies
> around 50-100 connections, POP3 around 10-20, with an added 5 LMTP
> connections for insertions.
Yuck, are you serious? Maybe splitting the table across multiple spindles would help.
I can try a "bare iron" test machine today. What is your "blessed" or preferred Linux Distro where you do your development and regression testing? I have no distro religious issues, though being a control freak, I will usually run "home" to the minimalistic comfort of Slackware.
> That said: dbmail will never be able to achieve the kind of numbers you
> mention for dovecot or panda. The single-instance storage used for both
> mime-blobs and header values, and fully indexed headers prevent it.
> DBMail insertion is not about creating simple files in directories.
I appreciate this difference. I realise that with Dovecot's scheme, they defer indexing until a user does an IMAP SEARCH TEXT/BODY, making the user wait a bit for that initial search before updating the index. I suppose to be fair, I should fire off a SEARCH for a known-to-be-non-existent string before closing the mailbox. It'd be trivial to update the script to do this since I am worried I'm not comparing "apples to apples" here.
I appreciate all of the replies I've received so far, especially from one of the developers.
DBmail mailing list
DBmail [at] dbmail