Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: exim: users

Nhash and Imap

 

 

exim users RSS feed   Index | Next | Previous | View Threaded


totty at dokom

Feb 1, 2007, 6:08 AM

Post #1 of 13 (1606 views)
Permalink
Nhash and Imap

Hi,

I want to use nhash to split up the users maildirs into subfolders.
My problem is, that I haven`t got a clue how the imap daemon deals with the
new directory structure.
Can anyone help me out of this?


Many regards
Torsten


--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


askbill at conducive

Feb 1, 2007, 10:25 AM

Post #2 of 13 (1546 views)
Permalink
Re: Nhash and Imap [In reply to]

Torsten wrote:
> Hi,
>
> I want to use nhash to split up the users maildirs into subfolders.
> My problem is, that I haven`t got a clue how the imap daemon deals with the
> new directory structure.
> Can anyone help me out of this?
>
>
> Many regards
> Torsten
>
>

Your IMAP daemon's docs are your starting point.

'basics' - so long as the folders are under the appropriate Maildir for a given
user, with the same perms, AND the MUA is NOT limited to 'show subscribed
folders only' Dovecot, to name one, picks them up.

Courier-IMAP is a little pickier about where they are (under INBOX) and whether
they have a "." prefix.

Bincimap, GW-IMAP, Cyrus, and others I haven't had hands-on for several years.

In all cases, just configure Exim to name and drop with whatever the IMAP daemon
expects.

HTH,

Bill

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


totty at dokom

Feb 2, 2007, 1:17 AM

Post #3 of 13 (1548 views)
Permalink
Re: Nhash and Imap [In reply to]

Just to make at clear.
I want exim to store mails like this:
file = "/var/mail/${nhash_127:$local_part}/$local_part/INBOX"
So if I understood it right, Exim builds a hash from "$localpart" and builds
a directory structure like this:
/var/mail/0/totty or /var/mail/127/bill

How can the Imap daemon know where to find the maildir for Totty or Bill?
Is there a buildin hash tool or something like that?
I searched a lot, but I couldn`t get an answer how Courier, Cyrus or other
MUA`s handle that directory structure.

Thanks
Torsten


> Your IMAP daemon's docs are your starting point.
>
> 'basics' - so long as the folders are under the appropriate Maildir for a
> given
> user, with the same perms, AND the MUA is NOT limited to 'show subscribed
> folders only' Dovecot, to name one, picks them up.
>
> Courier-IMAP is a little pickier about where they are (under INBOX) and
> whether
> they have a "." prefix.
>
> Bincimap, GW-IMAP, Cyrus, and others I haven't had hands-on for several
> years.
>
> In all cases, just configure Exim to name and drop with whatever the IMAP
> daemon
> expects.
>
> HTH,
>
> Bill
>
> --
> ## List details at http://www.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://www.exim.org/eximwiki/
>


--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


wbh at conducive

Feb 2, 2007, 4:16 AM

Post #4 of 13 (1552 views)
Permalink
Re: Nhash and Imap [In reply to]

Torsten wrote:
> Just to make at clear.
> I want exim to store mails like this:
> file = "/var/mail/${nhash_127:$local_part}/$local_part/INBOX"
> So if I understood it right, Exim builds a hash from "$localpart" and builds
> a directory structure like this:
> /var/mail/0/totty or /var/mail/127/bill

Not hard. Just an extra level to what is already done.

Exim writes where it is told to write - fixed or 'calculated'.

>
> How can the Imap daemon know where to find the maildir for Totty or Bill?
> Is there a buildin hash tool or something like that?

The IMAP daemon reads where it is told to read - fixed or 'calculated'.

Both can be driven 100% from a file, table, or DB, 100% 'calculated', or any mix
of the two that their respective variables and string-manip tools can stand.

(NB - Exim and Dovecot play very well together on that score - similar conventions).

All you need to do is insure that Exim and the IMAP daemon either use the same
'hard coded' record or parts thereof, and/or perform the same calculation.

Inserting hash or other functions is not greatly different from the use of
$local_part and $domain as 'variables'.

Note that w/r the use of a 'computed' location - hash or otherwise - neither has
to understand it - merely be assured of generating the same one.

Or NOT [1]

> I searched a lot, but I couldn`t get an answer how Courier, Cyrus or other
> MUA`s handle that directory structure.
>

Much the same as any other, really. It is just an added level in an
already-multi-level dirtree.

> Thanks
> Torsten
>

HTH,

Bill


[1] CAVEAT: Not a general-purpose recommendation.

We use SQL selects, and Exim & Dovecot *may* be set to see the storage
differently, as their records are synced by choice.

Or not.

This allows us to share incoming across departmental or functional lines,
(Helpdesk, NOC), transparantly alias, forward, duplicate, auto-respond,
sub-sort, quarantine, 'hand over' the correspondence history of a departing
staff member to their replacement, and selectively archive - all w/o touching
the Exim configure or re-HUp'ing.







>
>> Your IMAP daemon's docs are your starting point.
>>
>> 'basics' - so long as the folders are under the appropriate Maildir for a
>> given
>> user, with the same perms, AND the MUA is NOT limited to 'show subscribed
>> folders only' Dovecot, to name one, picks them up.
>>
>> Courier-IMAP is a little pickier about where they are (under INBOX) and
>> whether
>> they have a "." prefix.
>>
>> Bincimap, GW-IMAP, Cyrus, and others I haven't had hands-on for several
>> years.
>>
>> In all cases, just configure Exim to name and drop with whatever the IMAP
>> daemon
>> expects.
>>
>> HTH,
>>
>> Bill
>>
>> --
>> ## List details at http://www.exim.org/mailman/listinfo/exim-users
>> ## Exim details at http://www.exim.org/
>> ## Please use the Wiki with this list - http://www.exim.org/eximwiki/
>>
>
>


--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


totty at dokom

Feb 2, 2007, 8:51 AM

Post #5 of 13 (1539 views)
Permalink
Re: Nhash and Imap [In reply to]

Thank you very much, Bill.

This was really what I was searching for.
I can`t remember "Dovecot", sounds good. I was testing Courier a few
month ago, so I give Dovecot a try and test this scenario.


P.s.: The reason why I was asking for this: We want to scale our
Mailsystem to support more users by using GFS (loadbalanced
active/active clustering) and migrate from POP3 to IMAP. RedHat support
was telling me that splitting up maildirs is recommended because of
performence issues. Let`s see.


Thanks
Torsten


Am Freitag, den 02.02.2007, 20:16 +0800 schrieb W B Hacker:
> Torsten wrote:
> > Just to make at clear.
> > I want exim to store mails like this:
> > file = "/var/mail/${nhash_127:$local_part}/$local_part/INBOX"
> > So if I understood it right, Exim builds a hash from "$localpart" and builds
> > a directory structure like this:
> > /var/mail/0/totty or /var/mail/127/bill
>
> Not hard. Just an extra level to what is already done.
>
> Exim writes where it is told to write - fixed or 'calculated'.
>
> >
> > How can the Imap daemon know where to find the maildir for Totty or Bill?
> > Is there a buildin hash tool or something like that?
>
> The IMAP daemon reads where it is told to read - fixed or 'calculated'.
>
> Both can be driven 100% from a file, table, or DB, 100% 'calculated', or any mix
> of the two that their respective variables and string-manip tools can stand.
>
> (NB - Exim and Dovecot play very well together on that score - similar conventions).
>
> All you need to do is insure that Exim and the IMAP daemon either use the same
> 'hard coded' record or parts thereof, and/or perform the same calculation.
>
> Inserting hash or other functions is not greatly different from the use of
> $local_part and $domain as 'variables'.
>
> Note that w/r the use of a 'computed' location - hash or otherwise - neither has
> to understand it - merely be assured of generating the same one.
>
> Or NOT [1]
>
> > I searched a lot, but I couldn`t get an answer how Courier, Cyrus or other
> > MUA`s handle that directory structure.
> >
>
> Much the same as any other, really. It is just an added level in an
> already-multi-level dirtree.
>
> > Thanks
> > Torsten
> >
>
> HTH,
>
> Bill
>
>
> [1] CAVEAT: Not a general-purpose recommendation.
>
> We use SQL selects, and Exim & Dovecot *may* be set to see the storage
> differently, as their records are synced by choice.
>
> Or not.
>
> This allows us to share incoming across departmental or functional lines,
> (Helpdesk, NOC), transparantly alias, forward, duplicate, auto-respond,
> sub-sort, quarantine, 'hand over' the correspondence history of a departing
> staff member to their replacement, and selectively archive - all w/o touching
> the Exim configure or re-HUp'ing.
>
>
>
>
>
>
>
> >
> >> Your IMAP daemon's docs are your starting point.
> >>
> >> 'basics' - so long as the folders are under the appropriate Maildir for a
> >> given
> >> user, with the same perms, AND the MUA is NOT limited to 'show subscribed
> >> folders only' Dovecot, to name one, picks them up.
> >>
> >> Courier-IMAP is a little pickier about where they are (under INBOX) and
> >> whether
> >> they have a "." prefix.
> >>
> >> Bincimap, GW-IMAP, Cyrus, and others I haven't had hands-on for several
> >> years.
> >>
> >> In all cases, just configure Exim to name and drop with whatever the IMAP
> >> daemon
> >> expects.
> >>
> >> HTH,
> >>
> >> Bill
> >>
> >> --
> >> ## List details at http://www.exim.org/mailman/listinfo/exim-users
> >> ## Exim details at http://www.exim.org/
> >> ## Please use the Wiki with this list - http://www.exim.org/eximwiki/
> >>
> >
> >
>
>


--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


wbh at conducive

Feb 2, 2007, 11:20 AM

Post #6 of 13 (1539 views)
Permalink
Re: Nhash and Imap [In reply to]

Torsten wrote:
> Thank you very much, Bill.
>
> This was really what I was searching for.
> I can`t remember "Dovecot", sounds good. I was testing Courier a few
> month ago, so I give Dovecot a try and test this scenario.
>

Dovecot has a couple of advantages, one of them being not insisting that all
folders hang under INBOX, or start with a dot.

Though they *can* do - but 'INBOX.Trash' always struck me as a bit anachronistic.

;-)

It also has a fair following here, so is a road well-traveled w/r Exim+Dovecot
integration.

Cyrus has a longer history, but, JMHO, is a bit of an odd-man-out w/r
integration. Different string handling, special character handling, et al than
some parts of Exim.

But plenty of docs, examples, and expertise.

>
> P.s.: The reason why I was asking for this: We want to scale our
> Mailsystem to support more users by using GFS (loadbalanced
> active/active clustering) and migrate from POP3 to IMAP. RedHat support
> was telling me that splitting up maildirs is recommended because of
> performence issues. Let`s see.
>

RedHat *expects* 'performance issues'?

Refreshing change, but I am otherwise not going to go there.

;-)

Bill


--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


totty at dokom

Feb 2, 2007, 12:37 PM

Post #7 of 13 (1541 views)
Permalink
Re: Nhash and Imap [In reply to]

> RedHat *expects* 'performance issues'?
>
> Refreshing change, but I am otherwise not going to go there.

This performence issues are only mentioned with GFS clustering (locking
a file and so on).
Do you know another way to set up a loadbalanced active/active
clustering with the same data (same storage)?
I think GFS (Global File System) is the only way to setup two or more
systems to read/write on the same data.
If not, tell me.
By the way this GFS is so "hot" that the RedHat guys are telling me that
they need 5 days to setup this "loadbalanced active/active clustering
with the same data". This is like a workshop (with knowledge transfer).
By the way, they don't configure my Exim or Imap, it`s only for setting
up GFS and clustering.
So, let`s see.


--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


wbh at conducive

Feb 2, 2007, 1:46 PM

Post #8 of 13 (1535 views)
Permalink
Re: Nhash and Imap [In reply to]

Torsten wrote:
>> RedHat *expects* 'performance issues'?
>>
>> Refreshing change, but I am otherwise not going to go there.
>
> This performence issues are only mentioned with GFS clustering (locking
> a file and so on).
> Do you know another way to set up a loadbalanced active/active
> clustering with the same data (same storage)?
> I think GFS (Global File System) is the only way to setup two or more
> systems to read/write on the same data.
> If not, tell me.
> By the way this GFS is so "hot" that the RedHat guys are telling me that
> they need 5 days to setup this "loadbalanced active/active clustering
> with the same data". This is like a workshop (with knowledge transfer).
> By the way, they don't configure my Exim or Imap, it`s only for setting
> up GFS and clustering.
> So, let`s see.
>
>

My first 'large scale' email relied on twinned IBM 3080, later 3090...
... but I have learned not to be a believer in clustering for MTA.

Too many eggs in one basket - even if it is a strong and well-proven basket.
Which IBM mainframes were, but RH GFS is not (not yet, anyway...)

WHEN, not IF, s**t breaks, I prefer it breaks in manageable chunks.

Unless you are taking on Yahoo, MSN, or Gmail, a by-domain split ordinarily
serves even for large multi-domain ISP's, 'coz there are seldom that many in any
single domain.tld (or division thereof).

YMMV

Bill Hacker

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


tk at i22data

Feb 2, 2007, 2:21 PM

Post #9 of 13 (1534 views)
Permalink
Re: Nhash and Imap [In reply to]

> My first 'large scale' email relied on twinned IBM 3080, later 3090...
> ... but I have learned not to be a believer in clustering for MTA.
>
> Too many eggs in one basket - even if it is a strong and well-proven
> basket.
> Which IBM mainframes were, but RH GFS is not (not yet, anyway...)
>
> WHEN, not IF, s**t breaks, I prefer it breaks in manageable chunks.
>
> Unless you are taking on Yahoo, MSN, or Gmail, a by-domain split
> ordinarily
> serves even for large multi-domain ISP's, 'coz there are seldom that many
> in any
> single domain.tld (or division thereof).

I absolutly agree with this. Be on the safe side with more managable units
than one big whole s**t cluster solution. We tried to have such solutions
and failed - back to the roots.


Tobias Kuhrmann


--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


renaud at llorien

Feb 2, 2007, 2:50 PM

Post #10 of 13 (1540 views)
Permalink
Re: Nhash and Imap [In reply to]

Torsten wrote:
>> RedHat *expects* 'performance issues'?
>>
>> Refreshing change, but I am otherwise not going to go there.
>
> This performence issues are only mentioned with GFS clustering (locking
> a file and so on).
> Do you know another way to set up a loadbalanced active/active
> clustering with the same data (same storage)?
> I think GFS (Global File System) is the only way to setup two or more
> systems to read/write on the same data.
> If not, tell me.

You may be interested in unison:
http://www.cis.upenn.edu/~bcpierce/unison/

This is not a real time replication system, of course. But this could
handle maildirs fairly well and in the improbable case a a total crash
of one of the nodes, you still have all data (nearly) up to date on the
other as they just use different storages. This could even be used to
replicate data for servers residing in different countries. This is not
without issues of course, but may be interesting to test.
Attachments: smime.p7s (3.21 KB)


totty at dokom

Feb 3, 2007, 6:51 AM

Post #11 of 13 (1559 views)
Permalink
Re: Nhash and Imap [In reply to]

Thanks guys for your time.
So we set this up and see how it works.
I tell you later how it was, if you are interested in this.

Many regards
Torsten


--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


t.baer at dokom

Feb 8, 2007, 9:19 AM

Post #12 of 13 (1525 views)
Permalink
Re: Nhash and Imap [In reply to]

Hi John,

so you "patched" your imapd with the exim nhash code?
Is there no other way to get the imapd to deal with the nhash (builtin)?

I read the dovecot imapd documentation and I found this:
"%H - take a 32bit hash of the variable and return it as hex. You can
also limit the hash value. For example %256Hu gives values 0..ff. You
might want padding also, so %2.256Hu gives 00..ff. This can be useful
for example in dividing users automatically to multiple partitions. Note
that if you're hashing usernames being in user [at] domai form, you probably
want to reverse the string to get better hash value variety, eg. %3RHu."

I have to test that.
Because of "locked directories" I was wondering that it is so hard to
find anything about hashing maildirs in the internet.

If you have time please send me your C code, maybe this is the only way
to do this hashing.

Thanks
Torsten

Linn, Dr John A. schrieb:
> We use nhash and integrate it with Washington IMAP code. I probably can
> still find the C code I cut out of exim years ago, if you want.
>
>
> It creates the structure
> var/spool/mail/hash/userid/inbox
>
> which I was told minimises locked directories.
>
> John Linn
>
> -----Original Message-----
> From: exim-users-bounces [at] exim [mailto:exim-users-bounces [at] exim]
> On Behalf Of Torsten
> Sent: 01 February 2007 14:09
> To: exim-users [at] exim
> Subject: [exim] Nhash and Imap
>
> Hi,
>
> I want to use nhash to split up the users maildirs into subfolders.
> My problem is, that I haven`t got a clue how the imap daemon deals with
> the
> new directory structure.
> Can anyone help me out of this?
>
>
> Many regards
> Torsten
>
>
>

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


totty at dokom

Aug 22, 2008, 11:15 AM

Post #13 of 13 (1095 views)
Permalink
Re: Nhash and Imap [In reply to]

Hi there,

here is a wee field report of an loadbalanced 2 node RHEL 4 CS GFS
active/active pop3/imap cluster:

First I must say it runs excellent.

Some details:
Loadbalancing: Nortel Alteon

Hardware:
Nodes:
- Intel Dual Core Xeon 5130 (2 x 2000MHz)
- 4G Ram

Shared Storage:
- SAS Raid 10

Software:
- running OS RHEL 4 with Clustersuite and Gfs
- running Exim for incoming mails
- running Dovecot (pop3(s) and imap(s)) for serving data to mua`s
- running Mysql as database: serving user accounts and maildirs
- 32k pop3/imap accounts
- 7 gigs of incoming data per day (from the daily exim statistic of
2008.08.21 - 22)
- 90k incoming messages per day (from the daily exim statistic of
2008.08.21 - 22)
- 98 gigs of mail data stored

What else can I say, it runs since 1 year without any unwanted downtime
and load problems.

Regards
Torsten



> My first 'large scale' email relied on twinned IBM 3080, later 3090...
> ... but I have learned not to be a believer in clustering for MTA.
>
> Too many eggs in one basket - even if it is a strong and well-proven
basket.
> Which IBM mainframes were, but RH GFS is not (not yet, anyway...)
>
> WHEN, not IF, s**t breaks, I prefer it breaks in manageable chunks.
>
> Unless you are taking on Yahoo, MSN, or Gmail, a by-domain split
ordinarily
> serves even for large multi-domain ISP's, 'coz there are seldom that
many in any
> single domain.tld (or division thereof).
>
> YMMV
>
> Bill Hacker


Torsten wrote:/
>> RedHat *expects* 'performance issues'?
>>
>> Refreshing change, but I am otherwise not going to go there.
>
> This performence issues are only mentioned with GFS clustering (locking
> a file and so on).
> Do you know another way to set up a loadbalanced active/active
> clustering with the same data (same storage)?
> I think GFS (Global File System) is the only way to setup two or more
> systems to read/write on the same data.
> If not, tell me.
> By the way this GFS is so "hot" that the RedHat guys are telling me that
> they need 5 days to setup this "loadbalanced active/active clustering
> with the same data". This is like a workshop (with knowledge transfer).
> By the way, they don't configure my Exim or Imap, it`s only for setting
> up GFS and clustering.
> So, let`s see. /

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

exim users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.