Mailing List Archive: DBMail: users

IMAP quick resync?

Index | Next | Previous | View Threaded

shane at time-travellers

Sep 14, 2009, 7:57 AM

Post #1 of 10 (414 views)
Permalink

IMAP quick resync?

Hello,

Can any of the developers give any indication of how painful it would be
to add support for quick synchronization?

http://www.rfc-archive.org/getrfc.php?rfc=4551
http://www.rfc-archive.org/getrfc.php?rfc=5162

I'm not asking for a feature (although it would be nice), but rather a
sense of the scale of the task.

It makes me sad to have to check the status of every message in a
mailbox to see if any changes were made. ;)

Cheers,

--
Shane

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

vulture at netvulture

Sep 14, 2009, 10:02 AM

Post #2 of 10 (395 views)
Permalink

Re: IMAP quick resync? [In reply to]

Shane,
Briefly looking at the two RFC's, i think we could do this, but it will
take some work. This would have to go into the 2.3.x branch. What client
are you using that is supposed to support CONDSTORE and QRESYNC?

Paul,
I think we can add a modseq bigint to the messages table. This value
would be copied from the mailboxes seq. So we update a message, update
the seq, copy seq to that modified message, repeat for other messages.
This should satisfy the CONDSTORE requirements. Getting to protocol
responses and handling correct would take a while to cover all the
scenarios.

-Jon

Shane Kerr wrote:
> Hello,
>
> Can any of the developers give any indication of how painful it would be
> to add support for quick synchronization?
>
> http://www.rfc-archive.org/getrfc.php?rfc=4551
> http://www.rfc-archive.org/getrfc.php?rfc=5162
>
> I'm not asking for a feature (although it would be nice), but rather a
> sense of the scale of the task.
>
> It makes me sad to have to check the status of every message in a
> mailbox to see if any changes were made. ;)
>
> Cheers,
>
> --
> Shane
>
> _______________________________________________
> DBmail mailing list
> DBmail[at]dbmail.org
> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
>
>

--
Scanned for viruses and dangerous content by MailScanner

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

tabris at tabris

Sep 14, 2009, 10:22 AM

Post #3 of 10 (395 views)
Permalink

Re: IMAP quick resync? [In reply to]

Jonathan Feally wrote:
> Shane,
> Briefly looking at the two RFC's, i think we could do this, but it will
> take some work. This would have to go into the 2.3.x branch. What client
> are you using that is supposed to support CONDSTORE and QRESYNC?
>
> Paul,
> I think we can add a modseq bigint to the messages table. This value
> would be copied from the mailboxes seq. So we update a message, update
> the seq, copy seq to that modified message, repeat for other messages.
> This should satisfy the CONDSTORE requirements. Getting to protocol
> responses and handling correct would take a while to cover all the
> scenarios.
>

Sounds possibly related to UIDPLUS as well, so this gets my
(inconsequential) vote.

Attachments:

signature.asc (0.25 KB)

shane at time-travellers

Sep 14, 2009, 10:55 AM

Post #4 of 10 (394 views)
Permalink

Re: IMAP quick resync? [In reply to]

Jon,

I'm actually working on a Python client to do more-or-less intelligent
backups of IMAP mailboxes. (Yes I know this already exists in various
forms, but I want something that maintains multiple versions of mailbox
for historical reasons, and yet also takes a minimum of space. My
current approach is to use a Maildir folder, using Unix hard links to
messages that have not changed state at each backup.)

It's in the Thunderbird trunk:

http://www.rumblingedge.com/2008/10/05/2008-10-05-thunderbird-trunk-builds/

That's pretty big - don't know when it will be released (maybe already).

IMP claims to support these, at least in their beta:

http://www.horde.org/imp/roadmap/

Some clients use Tinymail, which claims to support these:

http://tinymail.org/usedby.html
http://tinymail.org/trac/tinymail/wiki/WhatTinymailSupports?format=txt

And there is this client:

http://dave.cridland.net/acap/polymer.html

I suspect the iPhone will support it at some point too; just a guess,
but it makes sense.

--
Shane

On Mon, 2009-09-14 at 10:02 -0700, Jonathan Feally wrote:
> Shane,
> Briefly looking at the two RFC's, i think we could do this, but it will
> take some work. This would have to go into the 2.3.x branch. What client
> are you using that is supposed to support CONDSTORE and QRESYNC?
>
> Paul,
> I think we can add a modseq bigint to the messages table. This value
> would be copied from the mailboxes seq. So we update a message, update
> the seq, copy seq to that modified message, repeat for other messages.
> This should satisfy the CONDSTORE requirements. Getting to protocol
> responses and handling correct would take a while to cover all the
> scenarios.
>
> -Jon
>
> Shane Kerr wrote:
> > Hello,
> >
> > Can any of the developers give any indication of how painful it would be
> > to add support for quick synchronization?
> >
> > http://www.rfc-archive.org/getrfc.php?rfc=4551
> > http://www.rfc-archive.org/getrfc.php?rfc=5162
> >
> > I'm not asking for a feature (although it would be nice), but rather a
> > sense of the scale of the task.
> >
> > It makes me sad to have to check the status of every message in a
> > mailbox to see if any changes were made. ;)
> >
> > Cheers,
> >
> > --
> > Shane
> >
> > _______________________________________________
> > DBmail mailing list
> > DBmail[at]dbmail.org
> > http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
> >
> >
>
>

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

shane at time-travellers

Sep 14, 2009, 11:48 PM

Post #5 of 10 (392 views)
Permalink

Re: IMAP quick resync? [In reply to]

Ug... reading this in the light of morning and it is confused. :-P

When I say "it's in" I mean CONDSTORE support is in the following
clients!

> It's in the Thunderbird trunk:
>
> http://www.rumblingedge.com/2008/10/05/2008-10-05-thunderbird-trunk-builds/
>
> That's pretty big - don't know when it will be released (maybe already).
>
> IMP claims to support these, at least in their beta:
>
> http://www.horde.org/imp/roadmap/
>
> Some clients use Tinymail, which claims to support these:
>
> http://tinymail.org/usedby.html
> http://tinymail.org/trac/tinymail/wiki/WhatTinymailSupports?format=txt
>
> And there is this client:
>
> http://dave.cridland.net/acap/polymer.html
>
> I suspect the iPhone will support it at some point too; just a guess,
> but it makes sense.
>
> --
> Shane
>
> On Mon, 2009-09-14 at 10:02 -0700, Jonathan Feally wrote:
> > Shane,
> > Briefly looking at the two RFC's, i think we could do this, but it will
> > take some work. This would have to go into the 2.3.x branch. What client
> > are you using that is supposed to support CONDSTORE and QRESYNC?
> >
> > Paul,
> > I think we can add a modseq bigint to the messages table. This value
> > would be copied from the mailboxes seq. So we update a message, update
> > the seq, copy seq to that modified message, repeat for other messages.
> > This should satisfy the CONDSTORE requirements. Getting to protocol
> > responses and handling correct would take a while to cover all the
> > scenarios.
> >
> > -Jon
> >
> > Shane Kerr wrote:
> > > Hello,
> > >
> > > Can any of the developers give any indication of how painful it would be
> > > to add support for quick synchronization?
> > >
> > > http://www.rfc-archive.org/getrfc.php?rfc=4551
> > > http://www.rfc-archive.org/getrfc.php?rfc=5162
> > >
> > > I'm not asking for a feature (although it would be nice), but rather a
> > > sense of the scale of the task.
> > >
> > > It makes me sad to have to check the status of every message in a
> > > mailbox to see if any changes were made. ;)
> > >
> > > Cheers,
> > >
> > > --
> > > Shane
> > >
> > > _______________________________________________
> > > DBmail mailing list
> > > DBmail[at]dbmail.org
> > > http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
> > >
> > >
> >
> >
>
> _______________________________________________
> DBmail mailing list
> DBmail[at]dbmail.org
> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
>

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

george.vieira at djtremors

Sep 15, 2009, 2:46 AM

Post #6 of 10 (391 views)
Permalink

Re: IMAP quick resync? [In reply to]

Hey Shane,

I'm looking for a way to "rsync" mail to a second server for mail migration.. I've worked out how to have multiple servers serving the same domain so you could have users in different locations and either mail server acting as a MX to the other for failover but need the ability to 'sync' the mail across when moving users to other mail stores/servers.

Since a mysqldump isn't really the right answer I thought a possible sync via a imap connection and store locally but it would need to be quicker if the mail already exists.
Not sure if this would be a good dbmail-rsync approach or just scripted but would make my project with dbmail quite handy since I'm working on a php based possibly enterprise level dbmail interface.

Sorry for getting off topic..

George

-----Original Message-----
From: dbmail-bounces[at]dbmail.org [mailto:dbmail-bounces[at]dbmail.org] On Behalf Of Shane Kerr
Sent: Tuesday, 15 September 2009 3:56 AM
To: DBMail mailinglist
Subject: Re: [Dbmail] IMAP quick resync?

Jon,

I'm actually working on a Python client to do more-or-less intelligent
backups of IMAP mailboxes. (Yes I know this already exists in various
forms, but I want something that maintains multiple versions of mailbox
for historical reasons, and yet also takes a minimum of space. My
current approach is to use a Maildir folder, using Unix hard links to
messages that have not changed state at each backup.)

It's in the Thunderbird trunk:

http://www.rumblingedge.com/2008/10/05/2008-10-05-thunderbird-trunk-builds/

That's pretty big - don't know when it will be released (maybe already).

IMP claims to support these, at least in their beta:

http://www.horde.org/imp/roadmap/

Some clients use Tinymail, which claims to support these:

http://tinymail.org/usedby.html
http://tinymail.org/trac/tinymail/wiki/WhatTinymailSupports?format=txt

And there is this client:

http://dave.cridland.net/acap/polymer.html

I suspect the iPhone will support it at some point too; just a guess,
but it makes sense.

--
Shane

On Mon, 2009-09-14 at 10:02 -0700, Jonathan Feally wrote:
> Shane,
> Briefly looking at the two RFC's, i think we could do this, but it will
> take some work. This would have to go into the 2.3.x branch. What client
> are you using that is supposed to support CONDSTORE and QRESYNC?
>
> Paul,
> I think we can add a modseq bigint to the messages table. This value
> would be copied from the mailboxes seq. So we update a message, update
> the seq, copy seq to that modified message, repeat for other messages.
> This should satisfy the CONDSTORE requirements. Getting to protocol
> responses and handling correct would take a while to cover all the
> scenarios.
>
> -Jon
>
> Shane Kerr wrote:
> > Hello,
> >
> > Can any of the developers give any indication of how painful it would be
> > to add support for quick synchronization?
> >
> > http://www.rfc-archive.org/getrfc.php?rfc=4551
> > http://www.rfc-archive.org/getrfc.php?rfc=5162
> >
> > I'm not asking for a feature (although it would be nice), but rather a
> > sense of the scale of the task.
> >
> > It makes me sad to have to check the status of every message in a
> > mailbox to see if any changes were made. ;)
> >
> > Cheers,
> >
> > --
> > Shane
> >
> > _______________________________________________
> > DBmail mailing list
> > DBmail[at]dbmail.org
> > http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
> >
> >
>
>

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

george.vieira at djtremors

Sep 17, 2009, 2:15 AM

Post #7 of 10 (386 views)
Permalink

Re: IMAP quick resync? [In reply to]

dbmail at tech

Sep 17, 2009, 2:37 AM

Post #8 of 10 (385 views)
Permalink

Re: IMAP quick resync? [In reply to]

George Vieira wrote:
> Hey Shane,
>
> I'm looking for a way to "rsync" mail to a second server for mail migration.. I've worked out how to have multiple servers serving the same domain so you could have users in different locations and either mail server acting as a MX to the other for failover but need the ability to 'sync' the mail across when moving users to other mail stores/servers.
>
> Since a mysqldump isn't really the right answer I thought a possible sync via a imap connection and store locally but it would need to be quicker if the mail already exists.
> Not sure if this would be a good dbmail-rsync approach or just scripted but would make my project with dbmail quite handy since I'm working on a php based possibly enterprise level dbmail interface.
>
>
Again, off topic but...

http://www.maatkit.org/doc/mk-table-sync.html

Is an option - it isn't great with large tables though. You could just
use replication? (obviously wouldn't be email user/domain level aware)

Ideally, need to be able to partition users/domains between servers -
could probably achieve something with a modified imap front end which
decides which backend to use. (or a complicated mysql proxy route?)

S
_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

george.vieira at djtremors

Sep 17, 2009, 5:38 AM

Post #9 of 10 (385 views)
Permalink

Re: IMAP quick resync? [In reply to]

Yeah I got dbmail configured for multiple servers hosting the same domain (or others) but need an option to move users to another server hence the question came up during this post.

I'll stick to my migration script which uses the dbmail mailbox2dbmail command. Works but only good once as if the source deletes an email, it stays on the receiving side.. but I guess you wouldn't be switching back to the source again after that anyway..

GV

-----Original Message-----
From: dbmail-bounces[at]dbmail.org [mailto:dbmail-bounces[at]dbmail.org] On Behalf Of Simon Gray
Sent: Thursday, 17 September 2009 7:37 PM
To: DBMail mailinglist
Subject: Re: [Dbmail] IMAP quick resync?

George Vieira wrote:
> Hey Shane,
>
> I'm looking for a way to "rsync" mail to a second server for mail migration.. I've worked out how to have multiple servers serving the same domain so you could have users in different locations and either mail server acting as a MX to the other for failover but need the ability to 'sync' the mail across when moving users to other mail stores/servers.
>
> Since a mysqldump isn't really the right answer I thought a possible sync via a imap connection and store locally but it would need to be quicker if the mail already exists.
> Not sure if this would be a good dbmail-rsync approach or just scripted but would make my project with dbmail quite handy since I'm working on a php based possibly enterprise level dbmail interface.
>
>
Again, off topic but...

http://www.maatkit.org/doc/mk-table-sync.html

Is an option - it isn't great with large tables though. You could just
use replication? (obviously wouldn't be email user/domain level aware)

Ideally, need to be able to partition users/domains between servers -
could probably achieve something with a modified imap front end which
decides which backend to use. (or a complicated mysql proxy route?)

S
_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

vulture at netvulture

Sep 17, 2009, 9:32 AM

Post #10 of 10 (384 views)
Permalink

Re: IMAP quick resync? [In reply to]

George Vieira wrote:
> Yeah I got dbmail configured for multiple servers hosting the same domain (or others) but need an option to move users to another server hence the question came up during this post.
>
DBMail Hyrda - (2.5.x) is planning on a split front-end, multi-back-end
setup to accomplish just what you are doing.

The current plan is to have a front-end database containing user and
mailbox schema tables. (This front-end could be circular replicated as
data here doesn't change much)
The back-end would contain messages for mailboxes that were assigned to
it. The mailboxes would be defined with a unique_id on the front end
with a pointer to a back-end-database. While we will probably target
having a single back-end assigned to all mailboxes of a user. Public
folders and other user folders may not be on the same back-end, so
access to those mailboxes would require the daemons to cross-connect to
all the back-ends. If a back-end is down when a select is done, then a
temporary error will be returned.

Migration tools would be part of this setup, allowing single mailboxes,
or all mailboxes owned by one user to be moved to a different back-end.

Database wise, this is a simple task, but coding wise requires some
major work as we have it coded to use a connection out of a single pool
and no concept of which server has what.

-Jon

--
Scanned for viruses and dangerous content by MailScanner

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com